{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "9qO--YcmIOzj"
   },
   "source": [
    "# TPU Fundamentals\n",
    "\n",
    "This codelab will walk you through the creation of a simple model using the TF-TPU programming primitives.  We also demonstrate how to use the TPUEstimator and Keras APIs to simplify common training tasks.\n",
    "\n",
    "For most models, we recommend using the high-level APIs, but understanding the underlying TPU programming model is important for debugging or for advanced/custom training scenarios."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 221
    },
    "colab_type": "code",
    "id": "Wc7mMgRgI3mY",
    "outputId": "8d720594-3165-4975-836c-4a9534732d71"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[_DeviceAttributes(/job:tpu_worker/replica:0/task:0/device:CPU:0, CPU, -1, 17773598204717337066),\n",
       " _DeviceAttributes(/job:tpu_worker/replica:0/task:0/device:XLA_CPU:0, XLA_CPU, 17179869184, 7715168056069560432),\n",
       " _DeviceAttributes(/job:tpu_worker/replica:0/task:0/device:XLA_GPU:0, XLA_GPU, 17179869184, 9710519241479157132),\n",
       " _DeviceAttributes(/job:tpu_worker/replica:0/task:0/device:TPU:0, TPU, 17179869184, 1102187703362188373),\n",
       " _DeviceAttributes(/job:tpu_worker/replica:0/task:0/device:TPU:1, TPU, 17179869184, 5891516886372362302),\n",
       " _DeviceAttributes(/job:tpu_worker/replica:0/task:0/device:TPU:2, TPU, 17179869184, 16333408726241456713),\n",
       " _DeviceAttributes(/job:tpu_worker/replica:0/task:0/device:TPU:3, TPU, 17179869184, 14387177117624451294),\n",
       " _DeviceAttributes(/job:tpu_worker/replica:0/task:0/device:TPU:4, TPU, 17179869184, 18168976666150053870),\n",
       " _DeviceAttributes(/job:tpu_worker/replica:0/task:0/device:TPU:5, TPU, 17179869184, 6826354838121806651),\n",
       " _DeviceAttributes(/job:tpu_worker/replica:0/task:0/device:TPU:6, TPU, 17179869184, 1793642356151583892),\n",
       " _DeviceAttributes(/job:tpu_worker/replica:0/task:0/device:TPU:7, TPU, 17179869184, 96424228969095596),\n",
       " _DeviceAttributes(/job:tpu_worker/replica:0/task:0/device:TPU_SYSTEM:0, TPU_SYSTEM, 17179869184, 8807202679840317663)]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import numpy as np\n",
    "import six\n",
    "import tensorflow as tf\n",
    "import time\n",
    "import os\n",
    "\n",
    "WORKER_NAME = \"laktpu\" #@param {type:\"string\"}\n",
    "\n",
    "TPU_WORKER = tf.contrib.cluster_resolver.TPUClusterResolver(\n",
    "    WORKER_NAME\n",
    ").get_master()\n",
    "\n",
    "session = tf.Session(TPU_WORKER)\n",
    "session.list_devices()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "1nO8m-SxIlNo"
   },
   "source": [
    "## What makes TPU modeling special?\n",
    "\n",
    "Writing TensorFlow models for the TPU differs from the CPU and GPU modeling you may be familiar with in a few important ways:\n",
    "\n",
    "Model functions are just-in-time compiled to run on the device.\n",
    "Almost all models are replicated by default, using synchronous in-graph replication (go/tf-strong-sync).\n",
    "An early goal of the TPU program was the ability to scale up. Individual TPU cores are powerful (~70 TFlops per core in the DragonFish generation), but the specialized network and hardware transfer support truly distinguish TPUs from other devices. The interconnect between TPU chips is an order of magnitude faster and lower-latency than the normal datacenter network fabric. This allows us to build models that scale up transparently using batch parallelism and synchronous replication.\n",
    "\n",
    "The TF-TPU programming model is designed to help your TF model take advantage of this scaling ability. Some models can scale to the size of an entire TPU pod (2048 cores!). To utilize TPUs effectively we need to make a few changes to how we develop our models, which we'll cover below."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Ejy438SBIlVW"
   },
   "source": [
    "## tpu.rewrite and model functions\n",
    "\n",
    "We refer to the portion of a TF model that runs on the TPU as a \"model function\". Most programs have a single model function which computes the training step for your model. Explicitly encapsulating the TPU related logic in a model function allows the TPU software stack to compile your model and implicitly replicate it across multiple TPU cores in a cluster. Let's start with a simple model function that adds two tensors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "r28ozZedINJZ"
   },
   "outputs": [],
   "source": [
    "def add(x, y):\n",
    "  return x + y"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "lA5S5EnuIyxo"
   },
   "source": [
    "This model takes 2 arguments and adds them together. Obviously this isn't a terribly interesting use of a TPU but we can run this model on the TPU nevertheless! We first need to wrap our model with a call to tpu.rewrite:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "rmNDNCF5Iv_B"
   },
   "outputs": [],
   "source": [
    "x = tf.placeholder(name='x', dtype=tf.float32)\n",
    "y = tf.placeholder(name='y', dtype=tf.float32)\n",
    "tpu_add = tf.contrib.tpu.rewrite(add, [x, y])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "MEFqNBPYI_zH"
   },
   "source": [
    "What's happening here? The tpu.rewrite call returns a version of our original function which is ready to be executed on the TPU. Now we can run it on our device!\n",
    "\n",
    "Note the initialize_system call. These are necessary to reset the TPU hardware. In the future, this initialization should occur automatically, but for now remember that you'll need to call them to initialize your session."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "YoQb33EBJAYZ",
    "outputId": "1d1501d1-a820-4b0e-ae9f-d47db47a2c8d"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "b'\\n\\x03\\x02\\x02\\x02\\x10\\x01\\x18\\x08\"\\x18\\x00\\x00\\x00\\x00\\x00\\x01\\x00\\x01\\x00\\x00\\x01\\x01\\x01\\x00\\x00\\x01\\x00\\x01\\x01\\x01\\x00\\x01\\x01\\x01'"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "session.run(tf.contrib.tpu.initialize_system())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 51
    },
    "colab_type": "code",
    "id": "nW3sazRcOJo0",
    "outputId": "d46a51dd-c9b4-4048-9435-88133368e063"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Result of TPU computation: %s [array([ 1.,  2.,  3.,  4.,  5.,  6.,  7.,  8.,  9., 10., 11., 12., 13.,\n",
      "       14., 15., 16.], dtype=float32)]\n"
     ]
    }
   ],
   "source": [
    "z = session.run(tpu_add, {x: np.arange(16), y: np.ones(16)})\n",
    "print('Result of TPU computation: %s', z)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "M8uaFrTgJKJB"
   },
   "source": [
    "## Running Logistic Regression\n",
    "\n",
    "Our first example wasn't very exciting. After all, we're interested in using the TPU to train models, not add tensors together. Let's change our model to learn a simple classifier via logistic regression.\n",
    "\n",
    "We'll use the well known MNIST dataset (easily imported via tf.keras.datasets), and to keep things as simple as possible, we will use a single fully connected layer to make our predictions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "bqE_SYCuJWWZ"
   },
   "outputs": [],
   "source": [
    "IMAGE_SIZE = 28 * 28\n",
    "NUM_LABELS = 10\n",
    "\n",
    "(x_train, y_train), (x_test, y_test) = tf.keras.datasets.mnist.load_data()\n",
    "x_train.shape, y_train.shape\n",
    "y_train = y_train.astype(np.int32)\n",
    "y_test = y_test.astype(np.int32)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "pmJbSakFL_Yx"
   },
   "outputs": [],
   "source": [
    "def fit_batch(img, labels):\n",
    "  with tf.variable_scope('tpu', reuse=tf.AUTO_REUSE):\n",
    "    # flatten images\n",
    "    x = tf.reshape(img, [-1, IMAGE_SIZE])\n",
    "    \n",
    "    W = tf.get_variable('W', [28*28, 10])  # pylint: disable=invalid-name\n",
    "    b = tf.get_variable('b', [10], initializer=tf.zeros_initializer)\n",
    "    logits = tf.matmul(x, W) + b\n",
    "    print(img, logits, labels)\n",
    "    loss = tf.losses.sparse_softmax_cross_entropy(labels, logits)\n",
    "    optimizer = tf.train.AdamOptimizer(learning_rate=0.1)\n",
    "    optimizer = tf.contrib.tpu.CrossShardOptimizer(optimizer)\n",
    "\n",
    "    return loss, optimizer.minimize(loss, tf.train.get_or_create_global_step())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "NWyumYURJbBS"
   },
   "source": [
    "The only unusual part of our model above the use of `CrossShardOptimizer`: this wraps a standard TF optimizer and allows us to train on multiple cores simultaneously (we'll demonstrate that below).\n",
    "\n",
    "And now we can train our model!  Note we use `tpu.rewrite` again to create a TPU version of our computation.  We'll only use the first 1000 images from our training data for this example."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 139
    },
    "colab_type": "code",
    "id": "xD1_aDfSJffS",
    "outputId": "4f1359c8-f057-47be-d61c-8943a6dda60b"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tensor(\"replicated_input_0_1:0\", shape=(?, 28, 28), dtype=float32) Tensor(\"tpu/add:0\", shape=(?, 10), dtype=float32) Tensor(\"replicated_input_1_1:0\", shape=(?,), dtype=int32)\n",
      "loss = [232.20671]\n",
      "loss = [431.7398]\n",
      "loss = [155.93834]\n",
      "loss = [71.937035]\n",
      "loss = [38.307476]\n"
     ]
    }
   ],
   "source": [
    "images = tf.placeholder(name='images', dtype=tf.float32, shape=[None, 28, 28])\n",
    "labels = tf.placeholder(name='labels', dtype=tf.int32, shape=[None,])\n",
    "\n",
    "fit_on_tpu = tf.contrib.tpu.rewrite(fit_batch, [images, labels])\n",
    "\n",
    "session.run(tf.global_variables_initializer())\n",
    "for i in range(50):\n",
    "  loss = session.run(fit_on_tpu, {\n",
    "      images: x_train[:1000], labels: y_train[:1000]\n",
    "  })\n",
    "  if i % 10 == 0:\n",
    "    print('loss = %s' % loss)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "VoDS1CuLJdxq"
   },
   "source": [
    "Now we can sample predictions from our test set to see how well we're doing:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "5drMu5d-KAgi"
   },
   "outputs": [],
   "source": [
    "def predict(img):\n",
    "  with tf.variable_scope('tpu', reuse=tf.AUTO_REUSE):\n",
    "    # flatten images\n",
    "    x = tf.reshape(img, [-1, IMAGE_SIZE])\n",
    "    \n",
    "    W = tf.get_variable('W', [28*28, 10])  # pylint: disable=invalid-name\n",
    "    b = tf.get_variable('b', [10], initializer=tf.zeros_initializer)\n",
    "    logits = tf.matmul(x, W) + b\n",
    "    return tf.nn.softmax(logits)\n",
    "\n",
    "predict_on_tpu = tf.contrib.tpu.rewrite(predict, [images,])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 486
    },
    "colab_type": "code",
    "id": "Tr2ThERwSMeG",
    "outputId": "7de87c93-ac67-46f5-99c6-d9a415b1fc29"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAS4AAAHVCAYAAABPFRaZAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzt3Xd4FVX+x/HPuTchIRAgtEAACQihBDQQug2EYAERXFm7yOoqKLKua8F1dy3ruri6rmIXsK4VESzYEEVRQgtVgvRQgtKkBNLvnd8fiZGql8yZrPPj/XoeHpM7h/MdbvJ8POfM3DnGcRwBgJ8E/tcnAADHiuAC4DsEFwDfIbgA+A7BBcB3CC4AvkNwAfAdgguA7xBcAHwn6n99AgD8IyMw1LOP2kwPTzKRtiW4jlPJY6Yd0y9gztgBEf9SAV5jqgjAdwguAL5DcAHwHYILgO8QXABc2fn7nqrxZQOtfaWTCgd2UyCtvec1uaoIwJXYXWFlz2ylKEm7rtmtOzq8rYzqBQe1KXCKNbeohnrHlqjVtOuUcu18VzUJLgCu1Hhrrmq89dP3jzXqrftOSZYOuIEmqiCsGku/U+857youJ9p1TYILgFWl329VjclbD3t96zU99dAPbZT8/DqVuqzBGhcAz0U1b6bH//y4Jj3aT6Xffe+6P4ILgKeKBnTVq1+/qa4xRvUmZFrpk+AC4KmN5wRU08TokvUZ1vokuAB4JhAfrytO+0p7w4Xadn9La/2yOA/AM6vvTtX79Z/U+at/o5gP3N0CcSBGXAA8sefyHlp60TitLS3QvgeaWu2bERcA60xUlL5+4Em9kpek1y7KUMwSe6MtiREXAC+c3EaS9MT9QxVessJ69wQXAKuC7VM04vV31P65G1Tn5Tme1CC4AFj17fUJOi9ur5rOLJYcb570THABsKbwvG6acd6/Pa9DcAGwZsspQZ0QFad/7myvqBlZntXhqiIAa1qOydS5Yzp7Xsc4Hs1BAcArTBUB+A7BBcB3CC4AvsPiPICIZQSGerYoPj08KeLd0v/fB9exbDXPNvOAP/y/Dy4AVSfnvp4KxTpqkLpdmSdP1omfDVf8vOpKHDfbah2CC4AVu6a11jdpj1d8X+JI3/aZoFe6NNab089QaMVqa7VYnAdgxddpr1d8/fTuluqffYEk6bL477T6qvpWaxFcAFwr7ZsuSXpkV4rO73+pPjy1pWLP26r7d3QsO57gdkOygxFcAFzb16SaHtmVopmDOiq0fKVCu3ZpzT2dNLruAklS04/sRg3BBcC1Oi9l6tMO8Spdl1PxWualD6lmIEZ/+q6H4j+1+zBBgguAdbuv6KnagVhlFgW1+L5OCu3da7V/gguAdTs6l90+OWzmNao+dZ71/rkdAoBVxdObK7Ptv3Vy5nVq96e1CnlQg+ACYE2bBdH6d+O3JcWq2YXfeBJaElNFAJbsGtZT9yR+IUkaltPP01oEFwArThs9VzUDMZKkzOxWntYiuABYMbZR2aavfZYNVbvb1nhai+ACYFXt68MK7drlaQ0W5wFYNXXW5IO+77XoEu3YWksJDfI0N/3Vw9q3f2WUWt6WeUw1CC4Anprd6bWKr/OdYpU4YUnSuUuv0p7F9dXkq2P/HCPBBcCK1BdHyTkgUbIv++kRN6mzhsvZWEMt39onzVsmSUrQaiWoco+6IbgAWNHizwdP9wbelv7TMS21WovFeQC+Q3AB8B2CC4DvGMfxbLchAPAEIy4AvkNwAfAdgguA7xBcAHyHG1ABRCwjMNSzq3nTw5NMpG19E1zJY6ZF/IbljB0Q8RsAwH+YKgLwHYILgKdMeqo+yF2oTX/tZa1PgguAp7Z1raVShRS3xd7yGMEFwFO7Tgppc2mR6k08tocF/hyCC4BnnFPSNGvgw+o/60ar/RJcADzT6fHFmluYpDaj1lntl+AC4Ilgahvd33ChdoZqKrR7j9W+CS4AnsjNqCdJysprbr1vgguAJ/a2L5EkLX48zXrfBBcAT6wfOF5dFlyqOi/Zu5r4I4ILgCdCTlhR7yR40rdvPqsIwD+iWjTX+D3bVfc5+6MtiREXAA+svi5J47L7eNY/wQXAunCzQhXsjvWsf4ILgFWF53XT0jOeUZMPg57VILgAWLVxkKMYE6Wa72R5VoPgAmBNsFYt3X7KB5Ikp7TUszpcVQRgTbioSNn5SeqX20XVtMGzOgQXAGucoiKt7CJPQ0tiqgjAhwguAL5jHMez3YYAwBOMuAD4DsEFwHcILgC+w+0QACKWERjq2aL49PCkiHegr9LgSh4z7Zj+0TljB0T8DwFw/GCqCMB3CC4A1gXr1Nb7uVlKWyQF26dY75/gAmBduEVTlTgh3dcwSzlD6lvvn8V5AFZFNWuqFs+u8bQGIy4A1mz8Wy8lvLFP/2o8q+K1mr22a9Nfe6ng/G7W6jDiAmBF9MzGWpHypEqckKSyp58uKg7rltafaEjaD5Kkge+kW6lFcAGw4vLGc1TihMqDS+owY4QazIhRzJ6Q7ugd0LKh47T5jl5q+s/ZrmsRXABcC6a20aAa8/TjSGvK/sZqe+MahfbulSS1WZ2ieYNi9eHIf6l/7G1Kvj9LTlFRpeuxxgXAtXC1n8ZAv9twtl7t26MitCQplL1K178wQo2D1bXw6kdk2rZ0VY/gAmBFtAlqYJN07TrlB5Vuzj3sePLbOxVtgoo2QW25x10tgguAaytHxlWsbR1NzgX1KtbAku5yV481LgCu/eW09456LKpZU+WlJ+np4U9KkuYVxcoUu9sBiOAC4Knsexppef/HJUmT9yXqqVuGKnbFPFd9ElwAPBM9s7H+2Xhyxfcv5PZS7HvuQksiuABY8OCbF+iSqx/R+7lZB611RZvgQd87Zx6+aF8ZLM4DcK3lGzuOemxeUayu2XCWfjNgmLV6BBcA10LZq5T60fVHPHb9cyO085RdCi9ZYa0eU0UAVqRcs0CnX3KDoq/aqo9S31C/W0bLMVLy4u36+Rsljh3BBcCaWq/NkV6Thqib4jVHkqyHlsRUEYAPEVwAfMc4jme7DQGAJxhxAfAdgguA7xBcAHyH4ALgO9zHBSBiGYGhnl3Nmx6eZCJtS3BZljxmWsQ/2JyxAyL+QQH4CVNFAL5DcAHwHYILgDWhPp01cvWaox4PtmllpQ7BBcCaDWfFqG5w31GPb/+3ncghuABYYaKr6cwzF/9sm98mZylYp7brWgQXACvGrpqlbvHrNTa9z1HbjE74ViY+3nUtgguAFU2jSvXKjQMU2rXriMejGjdSwFLkcB8XANd2/r6nXtqzV9GfZh21Tfa9zTQsp59C27a7rseIC4BrgcE7NPHVs496PJjaRv/t+4w2Ppwip6jIdT1GXABciWreTLPSXtWgAV2P2mbHv8LqEhNSjclzrdRkxAXAlfAPu/X37Z0V1bjREY9HNW+mr9Net7a+JTHiAuBSOC9Pn+S2Va/312vWMz0rXt/d3lHN5D3qkZSjsMJWaxJcAFxLuCdWt781Uw/clVnx2oKioEIKqEu1Ykl2nydAcAFwb94yXXHlaO1uHVPxUr3xZSGW+3aqsrq/YLUcwQXAiuDMhao38/DXCzbEK9Dd7nI6wQXAW46sr3FxVRGAp8KxZaG1PeT+/q0fEVwAPPXfs5/WiuKwMibeZq1PpooAPHXv+kHa/2QTnTB5trU+CS4A3uq7WTW02WqXTBUB+I5xHM92GwIATzDiAuA7BBcA3yG4APgOVxUBRCwjMNSzRfHp4UkRfxLbN8HF1vYAfsRUEYDvEFwAfIfgAuCp3Vf21MdbFmvTnb2s9UlwAfBMVJMk/f1vEyRJ2Tc8aa1fgguAZ7ad1Vz940okSZ0XXGStX4ILgCdK+qVr/n1PSZLyw8VqMGiltb4JLgCeWH9BsOLrC1cPtto3wQXAEwO6LpEk7QkXqOTuRKt9E1wAPPF4k7JdqzeXSoEvFlntm+AC4JkVxfm6tf/l1vsluABYVziwmyRpZUlDhVattd4/wQXAuoL6ZQvzt2Vd4En/BBcA64oG79aK4nw1nRDtSf8EFwCrgiknakHX/+rDfR0U/WmWJzUILgBWbfhNoqJNUDPObudZDYILgFUNz8yVJIW27/CsBsEFwBoTE6Pzk8puPHWKijyrQ3ABsCcU0rMrTvW8DMEFwBqntFTJY/ar3ddXeFrHN8+cB+APoTXrdcJQb2sw4gLgO8ZxPNttCAA8wYgLgO8QXAB8h+AC4DsEFwDf4XYIABHLCAz17Gre9PAkE2lbgsuy5DHTIv7B5owdEPEPCsBPmCoC8B2CC4DvEFwArIhq3kxJc+K1+onuCqa2OWq7YIMG2n1lT5mYmMrXqvTfBIAD3DtzstpEh3XmzkYKLV99xDbBBg102VcL1SN2im5Ydp20aHmlajHiAuBaVNMmOqlaUCfNuF4JA44cWpK04r5k/bbmNg1+/DY5lQwtieAC4FJU82Za9a8G6vLQjWo9bOFR2zk9T9aagc+o69gblfTgbFc1CS4Armx6tKZW935BTZ7/+RFUbu8a6jB7mBIfcxdaEsEFwCXHMSpxQgoXFB7xeCA+XqvHdddr1z2sE4Yus1KTxXkAVjScGaONeY1VPLFRxWvfn+bo3O6L9W7Sk5Ls7bHIiAuAK0lDsjU4/VzNWtJWM1Lf1oP/fFLf9Qnruz5htXq1UP9JKpsadpp7pbWajLgAuFb6/ValjNiqc0d0liSlaJ4kKXBSWwVkdN+ODmr+hz0qtVSPERcAz2y8K6iwHH3yj9NVummztX4JLgCe2HFtTy3t8bI2lhao+vZiq30TXAA8kZ+xT5J04eJrFPz86Pd3VQbBBcAT3/R6UXOKpIbnf2u9b4ILgCfCcnT1gmGSpGC9ugq2a22tb4ILgGfCoYC2jeqlzjO2a+M/qlnrl9shAHhmxenPK3y6o9Qvf6dWd+9XyFK/BBcATwzf0FeZc9uq7aNbdOL3KxUqPPJHgiqD4ALgie29dquV5li76fRArHEB8B3jOJ7tNgQAnmDEBcB3CC4AvkNwAfAdrioCiFhGYKhni+LTw5Mi3tmd4EKVSR4zLeJf+pyxAyL+Jcbxh6kiAN8huAD4DsEFwHcILgBWBevX05qXO8nExHhWg+ACYM22Ub10beZcfXvmBAXr1/OsDlcVAVix6rkuWnXWYwrIKCxp6rz3tKqkWBe8fLNa/mORwhafDsGIC4AVn/Z9RJL0wM52umjt2ZKklOhqGn/ZUwo0TrRai+ACYEVyVJxGbDpDX5xUXQXn5Ou1vEQFZHR6rNR+8kZFNW70y51EiOACYEVYjpY+07Hs6/379eb3XRWWo5AT1taieDmFRdZqEVwArBnyx89kunbUjmt76s1W72hFSYkk6eZG0xVq08xaHRbnAVixqqRYt9bL1u1TVygsRxetHaCC0Q005LWZGl5rk9aODujEOXZqMeICYMW1t96k9aWFCpqAAjIqOCdf4cXZGvvZeQrI6IEub1urxYgLgBU1J83VcN2sH36br8I9MUrZv0CS1GZMtvq2vkDTUyfrWUu1CC4A1tScNFc1Jx38WjgvT3undJBSpajGjVT63feu6zBVBOC5Bk9l6txLr9HUBdNk0lNd90dwAagSgS8WKSCjvH8UKBAf764vS+cEAL/o5bxG+rLjWyo9+URX/RBcAKrMm0POkCTtuNXd5xYJLgBVJrRitS5a11/vdZog9Tip0v1wVRFAlco7bYeu1qmSlla6D0ZcAHzHOI5nuw0BgCcYcQHwHYILgO8QXAB8h+AC4DvcDgEgYhmBoZ5dzZsenmQibUtwWZY8ZlrEP9icsQMi/kEB+AlTRQC+w4gLgOeiGiWquHWSJCl6Va5W3tFSdbKN6q4oVGDWomPvz/YJAsCP9lzeQzvPLdSYTh/pylofSJIm7jlBF8RPUcLQWEnSwCbpx9wvwQXAusDJ7TT+vfFqEJyvwCErUlfX3igp1lX/BBcA6/a3iFdisPphrz+9u6Ve2dD1oNdqa80x909wAXAtWKe23ln+mU758yglvJip6lPnaeDUdAXbtVZo5TopHKpoW5mgOhRXFQG4EoiNVdFbtdXm7etVf8ryg46FVqw+KLSs1bTeI4DjRjAhQSsfOlkftZuitveuU2jv3iqpS3ABqLQtl7fTyiFP6t39CQpt315ldQkuAJX27M2PSpJOq/6dQr07V1ldFucBVFq3mGiVOCElBGL17n+fVocZI1R7fqz2NXXUcup+SdKOk2ooceY2hVattVaX4AJQaS3e+71WDXxakhRtglrZb7zUr/zgFT+1mzfG6Kbsi1V34CordZkqAqi0Njcs0lnDrtUreY1V4hz96mG3GEdfdXrFWl1GXAAqzSktVfSnWXqtbZImXDhEoWijXrfM09hG8w9re+gd9G4w4gJgRY235qrWa3O05Ja0iteuyMlQWGHrtQguAFZFL1itHgsvkSS9nDxdAQVU5JQoY/lvrNUguABYFc7LU6MbC5VZFJQkTd5XXydPukkx/XOs1WCNC4B1pTkbNfqR65XXtUBt/7JDrTbMsdo/wQXAE4njZitRUqkHfTNVBOA7BBcA3zGO49luQwDgCUZcAHyH4ALgOwQXAN/hdggAEcsIDPVsUXx6eFLEO7sTXMep5DHTjukXMGfsgIh/qQCvMVUE4DsEFwDfIbgA+A7BBcB3CC4Anig+u6tWPd1NJy00apcVpe9v6mWtb4ILgBXhU9O07tU0Pbbha32Qu1C3P/6SEpL2aPLsbro/cbb6XTFHt61dphtWr9LmO9yFGLdDALDiHy+PV3q1oKTqumR9hvLOKlSD/SvVQNKI9H7aNrq5bn97lmYVNNbiUY9p8H/PV+mmzZWqxYgLgCuBGjW09t89lF4tqPlFjlLevF77BpYqvH9/RZuO8bkKRwVUL1Bdg2vsVkDubgtkxAXAld2DOuqzoQ9pRkGCxl4/TK0+maMfNyozUVEKtDlRE6bW1YMvvShJCpqAOs69VE22VX6DWIILgCt13l+uK4dfrumpk9X/hQkKOYfu6jOv4qtTlw5V3RtCSlqXLTefHSK4ALgSzstTTP88pTw1Uv3Tl2nVnobakFtfQzou0r8aLTiobcJVeSrdus11Tda4AFiRMnKecroVqFrGBrW+KkufbGxbcSynNF9p/xmlkIXQkhhxAfBIZtfnJFWTJF34r9uU9MRsa30z4gJg3ZZbe6m6KQutR3e1UqPnF1vtn+ACYNXqcd21+KbH9V0oX71HjdTHHWopnJ9vtQbBBcCaYL26WnTBI5Kk3l+NUtyUuZ7UIbgAWBFMSNBNc2eppomRJLX+/WrPahFcAKzYMait+seVVHx/4J3zthFcAKz4zS2fVtx82uq9EZ7WIrgAWPH82xllH+d5apRSRsz75b/gAvdxAbCi+V2zddZdaWome/drHY1xHM92GwIATzBVBOA7BBcA3yG4APgOwQXAd7iqCCBiGYGhnl3Nmx6eFPHznKs0uJLHTDumf3TO2AHuHkwN4P8lpooArCo8r5tMVJTW399TTefU1Pr7e2r9/T1VeF43azWYKgKwIli/nkJvVNfrrR/W1lC0OlYrv3t+2JeSpG2X5+u8Rreq3vhM17UILgBWrHr0BK1sO1FSnBoGpWs2naHN++tIkoImrGlt3tMbf3lQI1aMUuArdw8WZKoIwIo3ej0jSfqoIE69R43U1nOiFOi7SYG+m+T0+14pb16vE6NrquAvexXVKNFVLYILgBVp1aLUYfYwjWvVVnFT5iq0a9dPB8MhtfrjHL2el6AvOr6lteMauqpFcAGwJpgV/7PH75w/WJJ0Q+qXruoQXABcC5zUVqtKClV/acnPtkv4ItZOPSu9ADiurR5WR79bcYViPphfJfUILgCu/fGcaap1ztoqq0dwAXAlWKe24gMFVVqT4ALgyuarU3VZ/LaI2hadu0eSlB+u5qomwQWgyrzeaaIkacoDfV31Q3ABqBKlZ6arbXSMrs89RXXeWOiqL4ILgCuN/z1bT+9prqimTQ47Fj41TWtfTVPaIumTl8erzw0jlNOtQE5JsauafFYRgBWd3t+oBT+ccNBrY1s8q7RqZTGTVRxSzc++VchCLYILgGsvPDRQc/7+hNRgySFHolSqkJYUS5e/MVot9rp/MkRZrwDgUt3nMvXwza11c8Lqg15v+8XvVG1ZnJr+c7ZayE5oSQQXAEs+7RCvT9X5oNdayt3ja46GxXkAvkNwAfAd4ziebdoBAJ5gxAXAdwguAL5DcAHwHYILgO9wHxeAiGUEhnp2NW96eFLEO9cTXD6XPGZaxL9IOWMHRPyLAfyaMVUEYE0gLk5dF4f0fm6Wt3U87R3AcSMQF6dVz7bRXxssVFhhT2sxVQRgxbo7T1Z2n3G6bN052vmPFqom73b8YcQFwIrihqWSpKWzWqvaR95uU0ZwAbAiumax8sLFOmF6kee1mCoCsGL56c/pD1v6Kvj5kZ8nX3ROV+U1i1KDrL1yspa7qkVwAbDm61c7q5FmH/Ta2lc66dHur6ljta+UGIzRmpJSnf/WH3XiLXMqXYepIgDXgrVq6ZOCGmr0n59Cq6R/F922dplW9J6gAXGF+j4Uo4/zayu1WnU9OXiigu1TKl2P4ALg2vqbOmj0nEsqvg+2aqEPn39Kp8YWakZBnFpOuU53n3WxnmidoowV5+mM6vkqOKF2pesxVQTgWrt+q5X9eWtJkklP1djJE5U6c6RaX71C4cJCtdbcit191qxrpL/V7arYmcsqfbcXIy4ArjWN2y2Zsk+frRodo3bR0TrxskUKFxYe1ja6ZrH2l8Yc8VikCC4AroWdgOSUfRS2ceLuo945H2zVQstPf05ZO5q6qkdwAbCqztWFmlsUrXUP9FRUk6SDjrV7Y4O2hgoU+2hdVzUILgCuBFu10Om1v634vjR3ix7oN1jfXD5OjSbvVbBeWUjtvqKn/pb4tXq/eYvrO+tZnAfgSmjNer0wsJ+Wff6Y/nDuKZr3dCfV3FKqdpNH6dyeizVl6ecKKKBn9+zUb5v21Imq/P1bPyK4ALgWWrVWz+9tpv8kzZLunXXY8Q5fDVerm3dI2mKlHsEFwIqpvTto3PDB2t+iRB+f/YjO+ugmSVKbCYVKnr9UpRZrEVwArAht3aYmY7dJkm7UKUopf6yNF896ZnEegO8QXAB8h+AC4DvGcTzbbQgAPMGIC4DvEFwAfIfgAuA7BBcA3+EGVAARywgM9exq3vTwJBNpW4ILVSZ5zLSIf+lzxg6I+JcYxx+migB8h+AC4DsEFwDfIbgAWBfq3Vnv52bp4y2L9X5ulnZNa61AWnvtvqKnAvHxrvtncR6ANcVnddFHzz0lab5OuXO0ovPLrscUNA/o+1Ok6WMe1OCCm1Xjrbmu6hBcAKxwep6sfz71jCTpkrXnKuGFzIpjtVu1UNqkter63h+V4jK0JKaKACzZdWeB0mOkgd9eoPw/NDzo2O70RN3VMEvNPrFTi+ACYMXsTq9pc2mBAncmyFm0vOJ1ExOjVjdlK6CAqk+dZ6UWU0UAVoQV1obSWtKcpZKkYGobrbixtr497wlJ0oyCmtZqEVwArOkUs1+nLS2UJHWLe1t9qhdW7Gn9pyUXqqmWH/0vHwOmigCsGPTtEMWZarq13jLdWm+Zzqier9Nuv0GnLb5UktRwfJy1Woy4ANjRd7POHHK9tqWXjYeS/5Kp7S8X6du01zVxT7Liln9nbYsygguANXFT5ip5yk/ff3vmBIUV1hMrz1DSpmxrdZgqAvDEhjc7SpK6zb9SSUPshZZEcAHwyLvdn5YkmRkJ1vsmuAB4okVUrMIKK6rQ/rMHCS4AnggrrEd+aK964zN/ufExIrgAeOa5d/p50i/BBcATf/6+uxIXhDzpm9shAHjim/SwqsvOZxMPZRzHs007AMATTBUB+A7BBcB3CC4AvkNwAfAdrioCiFhGYKhnV/OmhydFvHt5lQbXsWzBLrENO4AjY6oIwHcILgC+Q3ABsC6qUaJWPd3toJ2sVz3dzV7/1noCcNwzMTFad09nPX7hBJ1RPV8lTkBhhTUr7VUpTRo0oquVOoy4AFiz8dZ0LbtinM6oni9JGr6hryd1CC4AVoRnNNOSkY9Jkqbsa6hBzXpoe6/dOr/lqTpn+EgtKAqq3zd5VmoRXACs+KDtVEWboAqdUt315sVSuOyRNk5RkaI/WaDLPx6hW+uu1frXTnZdi+ACYEVYYZU4IQ1fN0jJfz38qacpI+epxAnpsvbzXdciuAC4FtUkqeLrdZNae16P4ALg2qrRzSVJfW8YqcTHZv9s2zH1l2jfb3u4qkdwAXAtFBdWQAFVn/rzTzyNNkEFLMQOwQXAtZNOylFY4V9sV+KEImr3SwguAFVqQ2mxqm8vdtUHwQWgSp3//K0Kfr7QVR8EFwDXCs7YqoACmrjxK4X6dD5im/Wvn6Reiy7RCXf//OJ9JAguAFZ0mX+5EoPVtbl37GHH9v+mu97sPl517zv8WGUQXACsaHJnWFP219W7Vz2o1U90lySZ9FRtG9VLDz/0uNpVC0hzllqpxdMhAFgRWr5SL57dR888G9ZHAx/WBZtu1YRrH1OnmLKriGdnX6hq2mClFsEFwJrSdTmKuaSBRnT6g7Kef1Rt37tBktTi7bBiPl8qWw+sJ7gAWBXavl3Rn2zXoCZdlaKfbki1ucsGa1wAfMc4jme7DQGAJxhxAfAdgguA7xBcAHyHq4oAIpYRGOrZovj08KSId65nxAXAd3wz4koeMy3ipM8ZOyDi5AbgP4y4AHji4y2LlZhZy5O+CS4Annmp+ZfKH9Lder8EFwBPbTnd/soNwQXAU63+OMd6nwQXAE+t+Y+7rciOhOAC4DsEFwDfIbgA+A7BBcBTLM4DgAguAD5EcAHwHYILgKe4jwuAL3jx+cQDEVwArGtx2wpP+ye4AFiVP6S7Xmr+pSTptBuu8+R2CN88SBCAv5yVlKY4zfWkb0ZcAKyKmzJXZyWleVqD4ALgOwQXAN8xjuPZbkMA4AlGXAB8h+AC4DsEFwDfIbgA+I4nN6DWr1/fSU5O9qJrQFlZWTscx2nwvz6P41FGYKhnV/OmhydFvI+ZJ8GVnJysBQsWeNF1lUkeMy3itjljB3h4JjiUMWbD//o/+DOWAAAFwUlEQVQc8L/FVBGA7xBcAKwrPTNdg7J36v3cLH2Qu1Dv52YpfVFY6tbRSv98yBqAFasmdNEl6fMkSfc0fFaSFFZY7WZeq4bvxij+jTmSllmpRXABsGLVOc9oa6hAT+7spZQPr1O9zGjVm5ipE7XIei2CC4AVYYX15M5eyuoUUIq8vTjHGhcAKzr8d7TuabhIoT6dPa/FiAuAHUYKyGhnaqzqmnRJUsyC1Qrt3Wu9FMEFwIqxg19RWI7m3PGoAuWTud7LhqpoUqrqTcy0WovgAuBaVLOmGlQjS3dt66T3cjrImVNHgy7+Sje3/FSD792t8L2Ozr7iWkV9lmWnnpVeABzXSjdt1sAmZdPDJGVLkrIeCChLLfVseZutt8fouYnzdcfVI1wHGIvzAKpEkwdma25+K907Ybx2Xt3TVV8EF4AqM+nPZ2nanjQ9+Zdx2nh3r0r3Q3ABqDLV35mnJec109z8Vlr8+0cr3Q/BBaBKlW7O1bglfSquPFYGwQXAiq03/vLUL9g+Rfs/aqnPT31c5w2+qtK1uKoIwLWdV/fUvh4FSnzs8GNRzZpqw6UnSJImXPeY5hScqCF336q68yt/bxfBBcCKFb0naNH6sC7N/L2MpMS6e/V5x0kKaKHCchSQUasPr1P7u79T3c3ubkgluAC4Vm9iptqdeY0k6cWeE9UtxtG1m3qr3cxrFN5ZTS2nlEiSUj5foFIL9QguAFaceFnZ42vu1Y8fss7z5JE2EovzAHyI4ALgO8Zx7O82ZIzZLulYdmKpL2mH9ROpnF/TuUi/rvP5tZxLc7YnO755ElzHfBLGLHAcp8v/+jykX9e5SL+u8/k1nQuOb0wVAfgOwQXAd34twfXsLzepMr+mc5F+XefzazoXHMd+FWtcAHAsfi0jLgCIGMEFwHeqNLiMMWcbY1YaY9YYY8Yc4XiMMeaN8uNzjTHJHp1HM2PM58aYbGPMcmPMH47QprcxZo8xZnH5n795cS4H1Msxxiwrr3XYbpqmzLjy92apMcaTzeuMMW0O+DcvNsbsNcbcdEibKn1vgENV2WcVjTFBSU9IypC0WdJ8Y8y7juNkH9Dsakm7HMdpZYy5WNIDki7y4HRKJf3JcZyFxph4SVnGmOmHnIskzXIcZ6AH9Y+mj+M4R7vB8xxJrcv/dJf0VPl/rXIcZ6WkNKniZ5YracoRmlb1ewNUqMoRVzdJaxzHWec4TrGk1yWdf0ib8yW9WP71W5L6GmOM7RNxHOc7x3EWln+dJ2mFpCa261h2vqSXnDJzJNUxxjT2uGZfSWsdxzmWT0EAnqvK4GoiadMB32/W4WFR0cZxnFJJeyTV8/KkyqejnSTNPcLhnsaYJcaYD40xqV6ehyRH0ifGmCxjzLVHOB7J+2fbxZJeO8qxqnxvgIMc14+1McbUlDRZ0k2O4xy6T/hClX0mbp8x5lxJU1U2TfPKqY7j5BpjGkqaboz51nGcLz2s97OMMdUkDZJ0xxEOV/V7AxykKkdcuZKaHfB90/LXjtjGGBMlqbaknV6cjDEmWmWh9YrjOG8fetxxnL2O4+wr//oDSdHGmPpenEt5jdzy/25T2ZpSt0OaRPL+2XSOpIWO42w99EBVvzfAoaoyuOZLam2MaVH+f/OLJb17SJt3JQ0r//pCSZ85HtwhW75uNlHSCsdxHj5Km0Y/rq8ZY7qp7L3yKkRrlF8kkDGmhqT+kr45pNm7kq4sv7rYQ9Iex3G+8+J8yl2io0wTq/K9AY6kyqaKjuOUGmNGSfpYUlDSc47jLDfG3CtpgeM476osTF42xqyR9IPKws0Lp0i6QtIyY8zi8tf+LOmE8nN9WmXBOdIYUyqpQNLFXoRouURJU8qzIErSq47jfGSMGXHA+Xwg6VxJayTlSxru0bn8GJ4Zkq474LUDz6Uq3xvgMHzkB4DvcOc8AN8huAD4DsEFwHcILgC+Q3AB8B2CC4DvEFwAfOf/AEuwZlabq7VMAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x576 with 32 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from matplotlib import pyplot\n",
    "%matplotlib inline\n",
    "\n",
    "def plot_predictions(images, predictions):\n",
    "  f, axes = pyplot.subplots(16, 2)\n",
    "  for i in range(16):\n",
    "    axes[i, 0].bar(np.arange(10), predictions[i])\n",
    "    axes[i, 1].imshow(images[i])\n",
    "    axes[i, 1].axis('off')\n",
    "\n",
    "    if i != 15:\n",
    "      axes[i, 0].axis('off')\n",
    "    else:\n",
    "      axes[i, 0].get_yaxis().set_visible(False)\n",
    "  pyplot.gcf().set_size_inches(6, 8)  \n",
    "\n",
    "\n",
    "[predictions] = session.run(predict_on_tpu, {\n",
    "    images: x_test[:16],\n",
    "})\n",
    "plot_predictions(x_test[:16], predictions)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "B1fL6RDWVDnq"
   },
   "source": [
    "We can see our network has quickly converged towards mostly correct observations!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "JtwTIhtXVSwK"
   },
   "source": [
    "## Using Multiple TPU Cores with Batch Parallelism\n",
    "\n",
    "Up until this point our models have only been running on a single TPU core.  This is wasting a lot of the power of TPUs!\n",
    "\n",
    "Let's change that so we can take advantage of our entire TPU device (8 cores). Scaling up to multiple cores with TPUs uses a different mechanism than the asynchronous out of graph replication used in CPU deployments. With TPUs, the software stack handles the replication for you: this is important to keep in mind, as you must use the builtin TPU replication to take advantage of the specialized TPU network. Fortunately TPU replication is easy to use. \n",
    "\n",
    "We just replace `tpu.rewrite` with `tpu.batch_parallel`: the TPU system handles the rest!  We just need to make sure our batches are divisible by the number of cores (8)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 224
    },
    "colab_type": "code",
    "id": "LfQOOdO3TnYH",
    "outputId": "a73573db-4c7f-4a6d-a156-e0e5e269d38f"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tensor(\"replicated_input_0_3:0\", shape=(?, 28, 28), dtype=float32) Tensor(\"tpu_2/add:0\", shape=(?, 10), dtype=float32) Tensor(\"replicated_input_1_2:0\", shape=(?,), dtype=int32)\n",
      "loss = 166.694077 [array([163.0523 , 144.5979 , 164.59166, 162.33806, 165.97234, 186.97296,\n",
      "       169.37622, 176.65114], dtype=float32)]\n",
      "loss = 375.441833 [array([407.09512, 377.64536, 362.78525, 262.08066, 316.60074, 465.14825,\n",
      "       377.2031 , 434.97623], dtype=float32)]\n",
      "loss = 130.799042 [array([ 92.44463 ,  82.570015, 135.40613 , 104.12831 , 174.28824 ,\n",
      "        64.86359 , 175.1331  , 217.55833 ], dtype=float32)]\n",
      "loss = 42.901794 [array([75.510376, 12.122637, 34.710434, 15.478931, 36.888874, 27.621033,\n",
      "       82.39092 , 58.49115 ], dtype=float32)]\n",
      "loss = 13.986987 [array([10.711948 ,  6.426794 , 15.038361 ,  5.8103333,  1.3755245,\n",
      "       16.995008 , 35.589947 , 19.947977 ], dtype=float32)]\n"
     ]
    }
   ],
   "source": [
    "fit_multi_on_tpu = tf.contrib.tpu.batch_parallel(fit_batch, [images, labels], num_shards=8)\n",
    "\n",
    "session.run(tf.global_variables_initializer())\n",
    "for i in range(50):\n",
    "  loss = session.run(fit_multi_on_tpu, {\n",
    "      images: x_train[:1024], labels: y_train[:1024]\n",
    "  })\n",
    "  if i % 10 == 0:\n",
    "    print('loss = %f %s' % (np.mean(loss), loss))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 486
    },
    "colab_type": "code",
    "id": "ho8JwvZmXA_O",
    "outputId": "c0e265bc-b87d-4cc0-cb8e-b5b1bccad421"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAS4AAAHVCAYAAABPFRaZAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzt3Xd4FVX+x/HPuTchIRAgtEAACQihBDQQug2EgAoiuLJ20dVVVGRd14Lr7lrWdXF1XcUuYF0rIliwIYqihBaqBOmhBKVJCaTfO78/EiNVL5kzWebH+/U8PObODOc73Nzn4zln5s4xjuMIAPwk8L8+AQA4WgQXAN8huAD4DsEFwHcILgC+Q3AB8B2CC4DvEFwAfIfgAuA7Uf/rEwDgHxmBYZ591WZaeKKJ9FiC6ziVPHrqUX0Ac8YMjPhDBXiNoSIA3yG4APgOwQXAdwguAL5DcAFwZcfve6rGVw205tVOKhzUTYG09p7X5KoiAFdid4aVPaOVoiTtvGaX7uzwjjKqFxxwTIFTrDlFNdQ7tkStpl6nlGvnuapJcAFwpcbbc1Tj7Z9fP96ot+4/JVna7waaqIKwaiz5Xr1nv6e4nGjXNQkuAFaV/rBFNSZtOWT7lmt66uEf2yj5hbUqdVmDOS4Anotq3kxP/PkJTXysn0q//8F1ewQXAE8VDeyq1755S11jjOqNz7TSJsEFwFMbzg6oponRxesyrLVJcAHwTCA+Xpef9rX2hAu19YGW1tplch6AZ1bdk6oP6j+l81b9RjEfursFYn/0uAB4YvdlPbTkwrFaU1qgvQ82tdo2PS4A1pmoKH3z4FN6NS9Jr1+YoZjF9npbEj0uAF44uY0k6ckHhim8eLn15gkuAFYF26doxBvvqv3zN6rOK7M9qUFwAbDquxsSdG7cHjWdUSw53jzpmeACYE3hud00/dx/e16H4AJgzeZTgjohKk7/3NFeUdOzPKvDVUUA1rQcnalzRnf2vI5xPBqDAoBXGCoC8B2CC4DvEFwAfIfJeQARywgM82xSfFp4YsSrpRNcPpc8emrEH6ScMQMj/mAAxzKCC4A1Off3VCjWUYPUbco8eZJO/Pwqxc+trsSxs6zWIbgAWLFzamt9m/ZExesSR/quz3i92qWx3pp2hkLLV1mrxeQ8ACu+SXuj4udndrVU/+zzJUmXxn+vVVfWt1qL4ALgWmnfdEnSoztTdF7/S/TRqS0Ve+4WPbC9Y9n+BLcLkh2I4ALg2t4m1fTozhTNGNxRoWUrFNq5U6vv7aRRdedLkpp+bDdqCC4ArtV5OVOfdYhX6dqcim2ZlzysmoEY/en7Hor/zO7DBAkuANbturynagdilVkU1KL7Oym0Z4/V9gkuANZt71x2e+HwGdeo+pS51tvndggAVhVPa67Mtv/WyZnXqd2f1ijkQQ2CC4A1beZH69+N35EUq2YXfOtJaEkMFQFYsnN4T92b+KUkaXhOP09rEVwArDht1BzVDMRIkjKzW3lai+ACYMWYRmWLvvZZOkztbl/taS2CC4BVtW8IK7Rzp6c1mJwHYNWUmZMOeN1r4cXavqWWEhrkaU76a4cc3/7VkWp5e+ZR1SC4AHhqVqfXK37Od4pV4oQlSecsuVK7F9VXk6+P/nuMBBcAK1JfGilnv0TJvvTnR9ykzrxKzoYaavn2XmnuUklSglYpQZV71A3BBcCKFn8+cLg36Pb0n/dpidVaTM4D8B2CC4DvEFwAfMc4jmerDQGAJ+hxAfAdgguA7xBcAHyH4ALgO9yACiBiGYFhnl3NmxaeaCI91jfBlTx6asRvWM6YgRG/AQD8h6EiAN8huAB4yqSn6sPcBdr4117W2iS4AHhqa9daKlVIcZvtTY8RXAA8tfOkkDaVFqnehKN7WOAvIbgAeMY5JU0zBz2i/jNvstouwQXAM52eWKQ5hUlqM3Kt1XYJLgCeCKa20QMNF2hHqKZCu3ZbbZvgAuCJ3Ix6kqSsvObW2ya4AHhiT/sSSdKiJ9Kst01wAfDEukHj1GX+Jarzsr2riT8huAB4IuSEFfVugidt++a7igD8I6pFc43bvU11n7ff25LocQHwwKrrkjQ2u49n7RNcAKwLNytUwa5Yz9onuABYVXhuNy0541k1+SjoWQ2CC4BVGwY7ijFRqvlulmc1CC4A1gRr1dIdp3woSXJKSz2rw1VFANaEi4qUnZ+kfrldVE3rPatDcAGwxikq0oou8jS0JIaKAHyI4ALgO8ZxPFttCAA8QY8LgO8QXAB8h+AC4DvcDgEgYhmBYZ5Nik8LT4x4BfoqDa7k0VOP6h+dM2ZgxP8QAMcPhooAfIfgAmBdsE5tfZCbpbSFUrB9ivX2CS4A1oVbNFWJE9L9DbOUM7S+9faZnAdgVVSzpmrx3GpPa9DjAmDNhr/1UsKbe/WvxjMrttXstU0b/9pLBed1s1aHHhcAK6JnNNbylKdU4oQklT39dGFxWLe2/lRD036UJA16N91KLYILgBWXNZ6tEidUHlxSh+kj1GB6jGJ2h3Rn74CWDhurTXf2UtN/znJdi+AC4FowtY0G15irn3pak/c1VtubViu0Z48kqc2qFM0dHKuPrv+X+sferuQHsuQUFVW6HnNcAFwLV/u5D/S79Wfptb49KkJLkkLZK3XDiyPUOFhdC65+VKZtS1f1CC4AVkSboAY1SdfOU35U6abcQ/Ynv7ND0SaoaBPU5nvd1SK4ALi24vq4irmtI8k5v17FHFjS3e7qMccFwLW/nPb+EfdFNWuqvPQkPXPVU5KkuUWxMsXuVgAiuAB4KvveRlrW/wlJ0qS9iXr61mGKXT7XVZsEFwDPRM9orH82nlTx+sXcXop9311oSQQXAAseeut8XXz1o/ogN+uAua5oEzzgtXPmoZP2lcHkPADXWr65/Yj75hbF6pr1A/SbgcOt1SO4ALgWyl6p1I9vOOy+G54foR2n7FR48XJr9RgqArAi5Zr5Ov3iGxV95RZ9nPqm+t06So6Rkhdt0y/fKHH0CC4A1tR6fbb0ujRU3RSv2ZJkPbQkhooAfIjgAuA7xnE8W20IADxBjwuA7xBcAHyH4ALgOwQXAN/hPi4AEcsIDPPsat608EQT6bEEl2XJo6dG/IvNGTMw4l8UgJ8xVATgOwQXAN8huABYE+rTWdevWn3E/cE2razUIbgAWLN+QIzqBvcecf+2f9uJHIILgBUmuprOPHPRLx7z2+QsBevUdl2L4AJgxZiVM9Utfp3GpPc54jGjEr6TiY93XYvgAmBF06hSvXrTQIV27jzs/qjGjRSwFDncxwXAtR2/76mXd+9R9GdZRzwm+75mGp7TT6Gt21zXo8cFwLXAkO2a8NpZR9wfTG2j//Z9VhseSZFTVOS6Hj0uAK5ENW+mmWmvafDArkc8Zvu/wuoSE1KNSXOs1KTHBcCV8I+79PdtnRXVuNFh90c1b6Zv0t6wNr8l0eMC4FI4L0+f5rZVrw/WaeazPSu272rvqGbybvVIylFYYas1CS4AriXcG6s73p6hB+/OrNg2vyiokALqUq1Ykt3nCRBcANybu1SXXzFKu1rHVGyqN64sxHLfSVVW9xetliO4AFgRnLFA9WYcur1gfbwC3e1OpxNcALzlyPocF1cVAXgqHFsWWttC7u/f+gnBBcBT/z3rGS0vDitjwu3W2mSoCMBT960brH1PNdEJk2ZZa5PgAuCtvptUQ5usNslQEYDvGMfxbLUhAPAEPS4AvkNwAfAdgguA73BVEUDEMgLDPJsUnxaeGPE3sX0TXCxtD+AnDBUB+A7BBcB3CC4Antp1RU99snmRNt7Vy1qbBBcAz0Q1SdLf/zZekpR941PW2iW4AHhm64Dm6h9XIknqPP9Ca+0SXAA8UdIvXfPuf1qSlB8uVoPBK6y1TXAB8MS684MVP1+waojVtgkuAJ4Y2HWxJGl3uEAl9yRabZvgAuCJJ5qUrVq9qVQKfLnQatsEFwDPLC/O1239L7PeLsEFwLrCQd0kSStKGiq0co319gkuANYV1C+bmL8963xP2ie4AFhXNGSXlhfnq+n4aE/aJ7gAWBVMOVHzu/5XH+3toOjPsjypQXABsGr9bxIVbYKaflY7z2oQXACsanhmriQptG27ZzUILgDWmJgYnZdUduOpU1TkWR2CC4A9oZCeW36q52UILgDWOKWlSh69T+2+udzTOr555jwAfwitXqcThnlbgx4XAN8xjuPZakMA4Al6XAB8h+AC4DsEFwDfIbgA+A63QwCIWEZgmGdX86aFJ5pIjyW4LEsePTXiX2zOmIER/6IA/IyhIgDfIbgA+A7BBcCKqObNlDQ7Xque7K5gapsjHhds0EC7rugpExNT+VqV/psAsJ/7ZkxSm+iwztzRSKFlqw57TLBBA1369QL1iJ2sG5deJy1cVqla9LgAuBbVtIlOqhbUSdNvUMLAw4eWJC2/P1m/rblVQ564XU4lQ0siuAC4FNW8mVb+q4G6PHyTWg9fcMTjnJ4na/WgZ9V1zE1KemiWq5oEFwBXNj5WU6t6v6gmL/xyDyq3dw11mDVciY+7Cy2J4ALgkuMYlTghhQsKD7s/EB+vVWO76/XrHtEJw5ZaqcnkPAArGs6I0Ya8xiqe0Khi2w+nOTqn+yK9l/SUJHtrLNLjAuBK0tBsDUk/RzMXt9X01Hf00D+f0vd9wvq+T1itXivUf5LKhoad5lxhrSY9LgCulf6wRSkjtuicEZ0lSSmaK0kKnNRWARndv72Dmv9ht0ot1aPHBcAzG+4OKixHn/7jdJVu3GStXYILgCe2X9tTS3q8og2lBaq+rdhq2wQXAE/kZ+yVJF2w6BoFvzjy/V2VQXAB8MS3vV7S7CKp4XnfWW+b4ALgibAcXT1/uCQpWK+ugu1aW2ub4ALgmXAooK0je6nz9G3a8I9q1trldggAnll++gsKn+4o9avfqdU9+xSy1C7BBcATV63vq8w5bdX2sc068YcVChUe/itBlUFwAfDEtl671Eqzrd10uj/muAD4jnEcz1YbAgBP0OMC4DsEFwDfIbgA+A5XFQFELCMwzLNJ8WnhiRGv7E5wocokj54a8Yc+Z8zAiD/EOP4wVATgOwQXAN8huAD4DsEFwKpg/Xpa/UonmZgYz2oQXACs2Tqyl67NnKPvzhyvYP16ntXhqiIAK1Y+30UrBzyugIzCkqbMfV8rS4p1/iu3qOU/Fips8ekQ9LgAWPFZ30clSQ/uaKcL15wlSUqJrqZxlz6tQONEq7UILgBWJEfFacTGM/TlSdVVcHa+Xs9LVEBGp8dK7SdtUFTjRr/eSIQILgBWhOVoybMdy37et09v/dBVYTkKOWFtKYqXU1hkrRbBBcCaoX/8XKZrR22/tqfeavWulpeUSJJuaTRNoTbNrNVhch6AFStLinVbvWzdMWW5wnJ04ZqBKhjVQENfn6Gram3UmlEBnTjbTi16XACsuPa2m7WutFBBE1BARgVn5yu8KFtjPj9XARk92OUda7XocQGwoubEObpKt+jH3+arcHeMUvbNlyS1GZ2tvq3P17TUSXrOUi2CC4A1NSfOUc2JB24L5+Vpz+QOUqoU1biRSr//wXUdhooAPNfg6Uydc8k1mjJ/qkx6quv2CC4AVSLw5UIFZJT3jwIF4uPdtWXpnADgV72S10hfdXxbpSef6KodggtAlXlr6BmSpO23ufveIsEFoMqElq/ShWv76/1O46UeJ1W6Ha4qAqhSeadt19U6VdKSSrdBjwuA7xjH8Wy1IQDwBD0uAL5DcAHwHYILgO8QXAB8h9shAEQsIzDMs6t508ITTaTHElyWJY+eGvEvNmfMwIh/UQB+xlARgO/Q4wLguahGiSpunSRJil6ZqxV3tlSdbKO6ywsVmLnw6NuzfYIA8JPdl/XQjnMKNbrTx7qi1oeSpAm7T9D58ZOVMCxWkjSoSfpRt0twAbAucHI7jXt/nBoE5ylw0IzU1bU3SIp11T7BBcC6fS3ilRisfsj2Z3a11Kvrux6wrbZWH3X7BBcA14J1auvdZZ/rlD+PVMJLmao+Za4GTUlXsF1rhVaslcKhimMrE1QH46oiAFcCsbEqeru22rxzg+pPXnbAvtDyVQeElrWa1lsEcNwIJiRoxcMn6+N2k9X2vrUK7dlTJXUJLgCVtvmydlox9Cm9ty9BoW3bqqwuwQWg0p675TFJ0mnVv1eod+cqq8vkPIBK6xYTrRInpIRArN777zPqMH2Eas+L1d6mjlpO2SdJ2n5SDSXO2KrQyjXW6hJcACqtxfu/18pBz0iSok1QK/qNk/qV77z85+Pmjja6Ofsi1R200kpdhooAKq3NjQs1YPi1ejWvsUqcI1897Bbj6OtOr1qrS48LQKU5paWK/ixLr7dN0vgLhioUbdTr1rka02jeIccefAe9G/S4AFhR4+05qvX6bC2+Na1i2+U5GQorbL0WwQXAquj5q9RjwcWSpFeSpymggIqcEmUs+421GgQXAKvCeXlqdFOhMouCkqRJe+vr5Ik3K6Z/jrUazHEBsK40Z4NGPXqD8roWqO1ftqvV+tlW2ye4AHgicewsJUoq9aBthooAfIfgAuA7xnE8W20IADxBjwuA7xBcAHyH4ALgO9wOASBiGYFhnk2KTwtPjHhld4ILVSZ59NSIP/Q5YwZG/CHG8YehIgDfIbgA+A7BBcB3CC4AvkNwAfBE8VldtfKZbjppgVG7rCj9cHMva20TXACsCJ+aprWvpenx9d/ow9wFuuOJl5WQtFuTZnXTA4mz1O/y2bp9zVLduGqlNt3pLsS4HQKAFf94ZZzSqwUlVdfF6zKUN6BQDfatUANJI9L7aeuo5rrjnZmaWdBYi0Y+riH/PU+lGzdVqhY9LgCuBGrU0Jp/91B6taDmFTlKeesG7R1UqvC+fRXHdIzPVTgqoHqB6hpSY5cCcnebHj0uAK7sGtxRnw97WNMLEjTmhuFq9els/bRQmYmKUqDNiRo/pa4eevklSVLQBNRxziVqsrXyC8QSXABcqfPBMl1x1WWaljpJ/V8cr5Bz8Ko+cyt+OnXJMNW9MaSktdly890hgguAK+G8PMX0z1PK09erf/pSrdzdUOtz62tox4X6V6P5BxybcGWeSrdsdV2TOS4AVqRcP1c53QpULWO9Wl+ZpU83tK3Yl1Oar7T/jFTIQmhJ9LgAeCSz6/OSqkmSLvjX7Up6cpa1tulxAbBu8229VN2UhdZjO1up0QuLrLZPcAGwatXY7lp08xP6PpSv3iOv1ycdaimcn2+1BsEFwJpgvbpaeP6jkqTeX49U3OQ5ntQhuABYEUxI0M1zZqqmiZEktf79Ks9qEVwArNg+uK36x5VUvN7/znnbCC4AVvzm1s8qbj5t9f4IT2sRXACseOGdjLKv8zw9Uikj5v76X3CB+7gAWNH87lkacHeamsne/VpHYhzHs9WGAMATDBUB+A7BBcB3CC4AvkNwAfAdrioCiFhGYJhnV/OmhSdG/DznKg2u5NFTj+ofnTNmoLsHUwP4f4mhIgCrCs/tJhMVpXUP9FTT2TW17oGeWvdATxWe281aDYaKAKwI1q+n0JvV9UbrR7QlFK2O1crvnh/+lSRp62X5OrfRbao3LtN1LYILgBUrHztBK9pOkBSnhkHpmo1naNO+OpKkoAlrapv39eZfHtKI5SMV+NrdgwUZKgKw4s1ez0qSPi6IU++R12vL2VEK9N2oQN+Ncvr9oJS3btCJ0TVV8Jc9imqU6KoWwQXAirRqUeowa7jGtmqruMlzFNq58+ed4ZBa/XG23shL0Jcd39aasQ1d1SK4AFgTzIr/xf13zRsiSbox9StXdQguAK4FTmqrlSWFqr+k5BePS/gy1k49K60AOK6tGl5Hv1t+uWI+nFcl9QguAK798eypqnX2miqrR3ABcCVYp7biAwVVWpPgAuDKpqtTdWn81oiOLTpntyQpP1zNVU2CC0CVeaPTBEnS5Af7umqH4AJQJUrPTFfb6BjdkHuK6ry5wFVbBBcAVxr/e5ae2d1cUU2bHLIvfGqa1ryWprSF0qevjFOfG0cop1uBnJJiVzX5riIAKzp9sEHzfzzhgG1jWjyntGplMZNVHFLNz79TyEItgguAay8+PEiz//6k1GDxQXuiVKqQFhdLl705Si32uH8yRFmrAOBS3ecz9cgtrXVLwqoDtrf98neqtjROTf85Sy1kJ7QkgguAJZ91iNdn6nzAtpZy9/iaI2FyHoDvEFwAfMc4jmeLdgCAJ+hxAfAdgguA7xBcAHyH4ALgO9zHBSBiGYFhnl3NmxaeGPHK9QSXzyWPnhrxBylnzMCIPxjAsYyhIgBrAnFx6roopA9ys7yt42nrAI4bgbg4rXyujf7aYIHCCntai6EiACvW3nWysvuM1aVrz9aOf7RQNXm34g89LgBWFDcslSQtmdla1T72dpkygguAFdE1i5UXLtYJ04o8r8VQEYAVy05/Xn/Y3FfBLw7/PPmis7sqr1mUGmTtkZO1zFUtgguANd+81lmNNOuAbWte7aTHur+ujtW+VmIwRqtLSnXe23/UibfOrnQdhooAXAvWqqVPC2qo0X9+Dq2S/l10+5qlWt57vAbGFeqHUIw+ya+t1GrV9dSQCQq2T6l0PYILgGvrbu6gUbMvrngdbNVCH73wtE6NLdT0gji1nHyd7hlwkZ5snaKM5efqjOr5KjihdqXrMVQE4Fq7fquU/UVrSZJJT9WYSROUOuN6tb56ucKFhWqtORWr+6xe20h/q9tVsTOWVvpuL3pcAFxrGrdLMmXfPls5KkbtoqN14qULFS4sPOTY6JrF2lcac9h9kSK4ALgWdgKSU/ZV2MaJu45453ywVQstO/15ZW1v6qoewQXAqjpXF2pOUbTWPthTUU2SDtjX7s312hIqUOxjdV3VILgAuBJs1UKn1/6u4nVp7mY92G+Ivr1srBpN2qNgvbKQ2nV5T/0t8Rv1futW13fWMzkPwJXQ6nV6cVA/Lf3icf3hnFM095lOqrm5VO0mjdQ5PRdp8pIvFFBAz+3eod827akTVfn7t35CcAFwLbRyjV7Y00z/SZop3TfzkP0dvr5KrW7ZLmmzlXoEFwArpvTuoLFXDdG+FiX65KxHNeDjmyVJbcYXKnneEpVarEVwAbAitGWrmozZKkm6SacopfyxNl4865nJeQC+Q3AB8B2CC4DvGMfxbLUhAPAEPS4AvkNwAfAdgguA7xBcAHyHG1ABRCwjMMyzq3nTwhNNpMcSXKgyyaOnRvyhzxkzMOIPMY4/DBUB+A7BBcB3CC4AvkNwAbAu1LuzPsjN0iebF+mD3CztnNpagbT22nV5TwXi4123z+Q8AGuKB3TRx88/LWmeTrlrlKLzy67HFDQP6IdTpGmjH9KQgltU4+05ruoQXACscHqerH8+/awk6eI15yjhxcyKfbVbtVDaxDXq+v4fleIytCSGigAs2XlXgdJjpEHfna/8PzQ8YN+u9ETd3TBLzT61U4vgAmDFrE6va1NpgQJ3JchZuKxiu4mJUaubsxVQQNWnzLVSi6EiACvCCmt9aS1p9hJJUjC1jZbfVFvfnfukJGl6QU1rtQguANZ0itmn05YUSpK6xb2jPtULK9a0/tPiC9RUy478l48CQ0UAVgz+bqjiTDXdVm+pbqu3VGdUz9dpd9yo0xZdIklqOC7OWi16XADs6LtJZw69QVvTy/pDyX/J1LZXivRd2huasDtZccu+t7ZEGcEFwJq4yXOUPPnn19+dOV5hhfXkijOUtDHbWh2GigA8sf6tjpKkbvOuUNJQe6ElEVwAPPJe92ckSWZ6gvW2CS4AnmgRFauwwooqtP/sQYILgCfCCuvRH9ur3rjMXz/4KBFcADzz/Lv9PGmX4ALgiT//0F2J80OetM3tEAA88W16WNVl57uJBzOO49miHQDgCYaKAHyH4ALgOwQXAN8huAD4DlcVAUQsIzDMs6t508ITI169vEqD62iWYJdYhh3A4TFUBOA7BBcA3yG4AFgX1ShRK5/pdsBK1iuf6WavfWstATjumZgYrb23s564YLzOqJ6vEiegsMKamfaalCYNHtHVSh16XACs2XBbupZePlZnVM+XJF21vq8ndQguAFaEpzfT4usflyRN3ttQg5v10LZeu3Rey1N19lXXa35RUP2+zbNSi+ACYMWHbaco2gRV6JTq7rcuksJlj7RxiooU/el8XfbJCN1Wd43WvX6y61oEFwArwgqrxAnpqrWDlfzXQ596mnL9XJU4IV3afp7rWgQXANeimiRV/Lx2YmvP6xFcAFxbOaq5JKnvjdcr8fFZv3js6PqLtfe3PVzVI7gAuBaKCyuggKpP+eUnnkaboAIWYofgAuDaSSflKKzwrx5X4oQiOu7XEFwAqtT60mJV31bsqg2CC0CVOu+F2xT8YoGrNgguAK4VnLFFAQU0YcPXCvXpfNhj1r1xknotvFgn3PPLk/eRILgAWNFl3mVKDFbXpt6xh+zb95vueqv7ONW9/9B9lUFwAbCiyV1hTd5XV+9d+ZBWPdldkmTSU7V1ZC898vATalctIM1eYqUWT4cAYEVo2Qq9dFYfPftcWB8PekTnb7xN4699XJ1iyq4inpV9gappvZVaBBcAa0rX5ijm4gYa0ekPynrhMbV9/0ZJUot3wor5YolsPbCe4AJgVWjbNkV/uk2Dm3RVin6+IdXmKhvMcQHwHeM4nq02BACeoMcFwHcILgC+Q3AB8B2uKgKIWEZgmGeT4tPCEyNeuZ4eFwDf8U2PK3n01IiTPmfMwIiTG4D/0OMC4IlPNi9SYmYtT9omuAB45uXmXyl/aHfr7RJcADy1+XT7MzcEFwBPtfrjbOttElwAPLX6P+6WIjscgguA7xBcAHyH4ALgOwQXAE8xOQ8AIrgA+BDBBcB3CC4AnuI+LgC+4MX3E/dHcAGwrsXtyz1tn+ACYFX+0O56uflXkqTTbrzOk9shfPMgQQD+MiApTXGa40nb9LgAWBU3eY4GJKV5WoPgAuA7BBcA3zGO49lqQwDgCXpcAHyH4ALgOwQXAN8huAD4jic3oNavX99JTk72omlAWVlZ2x3HafC/Po/jUUZgmGdX86aFJ0a8jpknwZWcnKz58+d70fT/W8mjp0Z8bM6YgR6eybHAnLnyAAAFxklEQVTPGLP+f30O+N9iqAjAdwguANaVnpmuwdk79EFulj7MXaAPcrOUvjAsdetopX2+ZA3AipXju+ji9LmSpHsbPidJCiusdjOuVcP3YhT/5mxJS63UIrgAWLHy7Ge1JVSgp3b0UspH16leZrTqTcjUiVpovRbBBcCKsMJ6akcvZXUKKEXeXpxjjguAFR3+O0r3NlyoUJ/OnteixwXADiMFZLQjNVZ1TbokKWb+KoX27LFeiuACYMWYIa8qLEez73xMgfLBXO+lw1Q0MVX1JmRarUVwAXAtqllTDa6Rpbu3dtL7OR3kzK6jwRd9rVtafqYh9+1S+D5HZ11+raI+z7JTz0orAI5rpRs3aVCTsuFhkrIlSVkPBpSllnqu/Jgtd8To+QnzdOfVI1wHGJPzAKpEkwdnaU5+K903fpx2XN3TVVsEF4AqM/HPAzR1d5qe+stYbbinV6XbIbgAVJnq787V4nObaU5+Ky36/WOVbofgAlClSjflauziPhVXHiuD4AJgxZabfn3oF2yfon0ft9QXpz6hc4dcWelaXFUE4NqOq3tqb48CJT5+6L6oZk21/pITJEnjr3tcswtO1NB7blPdeZW/t4vgAmDF8t7jtXBdWJdk/l5GUmLdPfqi40QFtEBhOQrIqNVH16n9Pd+r7iZ3N6QSXABcqzchU+3OvEaS9FLPCeoW4+jajb3VbsY1Cu+oppaTSyRJKV/MV6mFegQXACtOvLTs8TX36acvWed58kgbicl5AD5EcAHwHeM49lcbMsZsk3Q0K7HUl7Td+olUzrF0LtKxdT7Hyrk0Z3my45snwXXUJ2HMfMdxuvyvz0M6ts5FOrbO51g6FxzfGCoC8B2CC4DvHCvB9dyvH1JljqVzkY6t8zmWzgXHsWNijgsAjsax0uMCgIgRXAB8p0qDyxhzljFmhTFmtTFm9GH2xxhj3izfP8cYk+zReTQzxnxhjMk2xiwzxvzhMMf0NsbsNsYsKv/zNy/OZb96OcaYpeW1DllN05QZW/7eLDHGeLJ4nTGmzX7/5kXGmD3GmJsPOqZK3xvgYFX2XUVjTFDSk5IyJG2SNM8Y857jONn7HXa1pJ2O47Qyxlwk6UFJF3pwOqWS/uQ4zgJjTLykLGPMtIPORZJmOo4zyIP6R9LHcZwj3eB5tqTW5X+6S3q6/L9WOY6zQlKaVPE7y5U0+TCHVvV7A1Soyh5XN0mrHcdZ6zhOsaQ3JJ130DHnSXqp/Oe3JfU1xhjbJ+I4zveO4ywo/zlP0nJJTWzXsew8SS87ZWZLqmOMaexxzb6S1jiOczTfggA8V5XB1UTSxv1eb9KhYVFxjOM4pZJ2S6rn5UmVD0c7SZpzmN09jTGLjTEfGWNSvTwPSY6kT40xWcaYaw+zP5L3z7aLJL1+hH1V+d4ABziuH2tjjKkpaZKkmx3HOXid8AUq+07cXmPMOZKmqGyY5pVTHcfJNcY0lDTNGPOd4zhfeVjvFxljqkkaLOnOw+yu6vcGOEBV9rhyJTXb73XT8m2HPcYYEyWptqQdXpyMMSZaZaH1quM47xy833GcPY7j7C3/+UNJ0caY+l6cS3mN3PL/blXZnFK3gw6J5P2z6WxJCxzH2XLwjqp+b4CDVWVwzZPU2hjTovz/5hdJeu+gY96TNLz85wskfe54cIds+bzZBEnLHcd55AjHNPppfs0Y001l75VXIVqj/CKBjDE1JPWX9O1Bh70n6Yryq4s9JO12HOd7L86n3MU6wjCxKt8b4HCqbKjoOE6pMWakpE8kBSU97zjOMmPMfZLmO47znsrC5BVjzGpJP6os3LxwiqTLJS01xiwq3/ZnSSeUn+szKgvO640xpZIKJF3kRYiWS5Q0uTwLoiS95jjOx8aYEfudz4eSzpG0WlK+pKs8OpefwjND0nX7bdv/XKryvQEOwVd+APgOd84D8B2CC4DvEFwAfIfgAuA7BBcA3yG4APgOwQXAd/4PshdmVjJhcO8AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x576 with 32 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "[predictions] = session.run(predict_on_tpu, {\n",
    "    images: x_test[:16], labels: y_train[:16]\n",
    "})\n",
    "\n",
    "plot_predictions(x_test[:16], predictions)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "_QyUkT7EWe-2"
   },
   "source": [
    "If you're paying close attention to the output on this run, you may notice that our print statement for loss is now reporting 8 values instead of a single scalar! tpu.batch_parallel concatenates the output from each core to return a result. We are thus seeing the loss computed by each individual TPU core. Our CrossShardOptimizer takes care of averaging our gradients, but we must average the loss on the CPU if we want to obtain a single scalar value:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "AFo30YsXTqYP"
   },
   "source": [
    "## Using Infeed and Outfeed\n",
    "\n",
    "In our previous examples, our TPU execution looked something like this:\n",
    "\n",
    "![alt text](https://screenshot.googleplex.com/j4PwT4FtE2n.png)\n",
    "\n",
    "Between each call to fit the TPU device is idle waiting for the CPU. If we could queue inputs and outputs, the TPU could work on new data while the CPU is working. Depending on our model, we may be able to entirely decouple the CPU and TPU training: the CPU feeds data and pulls out loss values, and the TPU runs the training independently as long as data is available:\n",
    "\n",
    "![queue based TPU execution](https://screenshot.googleplex.com/Oae4fq6kVm7.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "TjqB9BNlXgYd"
   },
   "outputs": [],
   "source": [
    "BATCH_SIZE = 1024\n",
    "\n",
    "def fit_batch_with_infeed():\n",
    "  \"\"\"Train one batch, reading from infeed and writing to outfeed.\"\"\"\n",
    "  \n",
    "  # each core will read 1/8 of the total batch size\n",
    "  images, labels = tf.contrib.tpu.infeed_dequeue_tuple(\n",
    "      dtypes=[tf.float32, tf.int32],\n",
    "      shapes=[(BATCH_SIZE // 8, 28, 28,), (BATCH_SIZE // 8,)],\n",
    "      name='infeed_dequeue')\n",
    "  loss, train_op = fit_batch(images, labels)\n",
    "  return tf.contrib.tpu.outfeed_enqueue_tuple((loss,)), train_op"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "z4_Tsh2OYD2_"
   },
   "source": [
    "Note we can still re-use our original training function!  We've just wrapped it in the logic to remove batches from the infeed and push our loss onto the outfeed.  We return both the minimize operation and the enqueue operation from our model function to ensure that both are run as part of the TPU program.\n",
    "\n",
    "Now let's look at the CPU side. Note that we need an enqueue/dequeue operation for each TPU core. We use the standard tf.device scope to assign our operations to a given core:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "colab": {
     "height": 286
    },
    "colab_type": "code",
    "id": "jAJZL-i6YEwW",
    "outputId": "d70ca13c-5eba-4a66-b7e4-a88c5d9f427b"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[[<tf.Operation 'InfeedEnqueueTuple' type=InfeedEnqueueTuple>,\n",
       "  <tf.Operation 'InfeedEnqueueTuple_1' type=InfeedEnqueueTuple>,\n",
       "  <tf.Operation 'InfeedEnqueueTuple_2' type=InfeedEnqueueTuple>,\n",
       "  <tf.Operation 'InfeedEnqueueTuple_3' type=InfeedEnqueueTuple>,\n",
       "  <tf.Operation 'InfeedEnqueueTuple_4' type=InfeedEnqueueTuple>,\n",
       "  <tf.Operation 'InfeedEnqueueTuple_5' type=InfeedEnqueueTuple>,\n",
       "  <tf.Operation 'InfeedEnqueueTuple_6' type=InfeedEnqueueTuple>,\n",
       "  <tf.Operation 'InfeedEnqueueTuple_7' type=InfeedEnqueueTuple>],\n",
       " [[<tf.Tensor 'OutfeedDequeueTuple:0' shape=() dtype=float32>],\n",
       "  [<tf.Tensor 'OutfeedDequeueTuple_1:0' shape=() dtype=float32>],\n",
       "  [<tf.Tensor 'OutfeedDequeueTuple_2:0' shape=() dtype=float32>],\n",
       "  [<tf.Tensor 'OutfeedDequeueTuple_3:0' shape=() dtype=float32>],\n",
       "  [<tf.Tensor 'OutfeedDequeueTuple_4:0' shape=() dtype=float32>],\n",
       "  [<tf.Tensor 'OutfeedDequeueTuple_5:0' shape=() dtype=float32>],\n",
       "  [<tf.Tensor 'OutfeedDequeueTuple_6:0' shape=() dtype=float32>],\n",
       "  [<tf.Tensor 'OutfeedDequeueTuple_7:0' shape=() dtype=float32>]]]"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from tensorflow.contrib.tpu.ops.gen_tpu_ops import infeed_enqueue_tuple, outfeed_dequeue_tuple\n",
    "\n",
    "def setup_feed(image_batch, label_batch):\n",
    "  \"\"\"Generate TF operations for CPU side infeed and outfeed.\"\"\"\n",
    "  infeed_ops = []\n",
    "  outfeed_ops = []\n",
    "  \n",
    "  # Split our input into 8 pieces and infeed each sub-batch  \n",
    "  infeed_batches = list(zip(tf.split(image_batch, 8), tf.split(label_batch, 8)))\n",
    "  \n",
    "  for i in range(8):\n",
    "    infeed_op = infeed_enqueue_tuple(\n",
    "      infeed_batches[i],\n",
    "      [b.shape for b in infeed_batches[i]],\n",
    "      device_ordinal=i\n",
    "    )\n",
    "    infeed_ops.append(infeed_op)\n",
    "    \n",
    "    outfeed_op = outfeed_dequeue_tuple(\n",
    "        dtypes=[tf.float32], shapes=[[]], device_ordinal=i)\n",
    "    outfeed_ops.append(outfeed_op)\n",
    "\n",
    "  return [infeed_ops, outfeed_ops]\n",
    "\n",
    "setup_feed(tf.placeholder(tf.float32, [1024, 28*28]),\n",
    "           tf.placeholder(tf.int32, [1024]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "colab": {
     "height": 654
    },
    "colab_type": "code",
    "id": "jUh_2F8qZCZY",
    "outputId": "93f68150-b060-4d9f-8a4a-31c8b0ba23ca"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tensor(\"infeed_dequeue_1:0\", shape=(128, 28, 28), dtype=float32) Tensor(\"tpu_4/add:0\", shape=(128, 10), dtype=float32) Tensor(\"infeed_dequeue_1:1\", shape=(128,), dtype=int32)\n",
      "priming infeed\n",
      "loss = 182.433533\n",
      "loss = 376.260010\n",
      "loss = 145.241806\n",
      "loss = 54.118515\n",
      "loss = 21.477819\n",
      "final loss = 7.697616\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAS4AAAHVCAYAAABPFRaZAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzt3Xd4FVX+x/HPuTchIRAgtEAACQihBDQQug2EYAERXFm7yOoqKLKua8F1dy3ruri6rmIXsK4VESzYEEVRQgtVgvRQgtKkBNLvnd8fiZGql8yZrPPj/XoeHpM7h/MdbvJ8POfM3DnGcRwBgJ8E/tcnAADHiuAC4DsEFwDfIbgA+A7BBcB3CC4AvkNwAfAdgguA7xBcAHwn6n99AgD8IyMw1LOP2kwPTzKRtiW4jlPJY6Yd0y9gztgBEf9SAV5jqgjAdwguAL5DcAHwHYILgO8QXABc2fn7nqrxZQOtfaWTCgd2UyCtvec1uaoIwJXYXWFlz2ylKEm7rtmtOzq8rYzqBQe1KXCKNbeohnrHlqjVtOuUcu18VzUJLgCu1Hhrrmq89dP3jzXqrftOSZYOuIEmqiCsGku/U+857youJ9p1TYILgFWl329VjclbD3t96zU99dAPbZT8/DqVuqzBGhcAz0U1b6bH//y4Jj3aT6Xffe+6P4ILgKeKBnTVq1+/qa4xRvUmZFrpk+AC4KmN5wRU08TokvUZ1vokuAB4JhAfrytO+0p7w4Xadn9La/2yOA/AM6vvTtX79Z/U+at/o5gP3N0CcSBGXAA8sefyHlp60TitLS3QvgeaWu2bERcA60xUlL5+4Em9kpek1y7KUMwSe6MtiREXAC+c3EaS9MT9QxVessJ69wQXAKuC7VM04vV31P65G1Tn5Tme1CC4AFj17fUJOi9ur5rOLJYcb570THABsKbwvG6acd6/Pa9DcAGwZsspQZ0QFad/7myvqBlZntXhqiIAa1qOydS5Yzp7Xsc4Hs1BAcArTBUB+A7BBcB3CC4AvsPiPICIZQSGerYoPj08KeLd0v/fB9exbDXPNvOAP/y/Dy4AVSfnvp4KxTpqkLpdmSdP1omfDVf8vOpKHDfbah2CC4AVu6a11jdpj1d8X+JI3/aZoFe6NNab089QaMVqa7VYnAdgxddpr1d8/fTuluqffYEk6bL477T6qvpWaxFcAFwr7ZsuSXpkV4rO73+pPjy1pWLP26r7d3QsO57gdkOygxFcAFzb16SaHtmVopmDOiq0fKVCu3ZpzT2dNLruAklS04/sRg3BBcC1Oi9l6tMO8Spdl1PxWualD6lmIEZ/+q6H4j+1+zBBgguAdbuv6KnagVhlFgW1+L5OCu3da7V/gguAdTs6l90+OWzmNao+dZ71/rkdAoBVxdObK7Ptv3Vy5nVq96e1CnlQg+ACYE2bBdH6d+O3JcWq2YXfeBJaElNFAJbsGtZT9yR+IUkaltPP01oEFwArThs9VzUDMZKkzOxWntYiuABYMbZR2aavfZYNVbvb1nhai+ACYFXt68MK7drlaQ0W5wFYNXXW5IO+77XoEu3YWksJDfI0N/3Vw9q3f2WUWt6WeUw1CC4Anprd6bWKr/OdYpU4YUnSuUuv0p7F9dXkq2P/HCPBBcCK1BdHyTkgUbIv++kRN6mzhsvZWEMt39onzVsmSUrQaiWoco+6IbgAWNHizwdP9wbelv7TMS21WovFeQC+Q3AB8B2CC4DvGMfxbLchAPAEIy4AvkNwAfAdgguA7xBcAHyHG1ABRCwjMNSzq3nTw5NMpG19E1zJY6ZF/IbljB0Q8RsAwH+YKgLwHYILgKdMeqo+yF2oTX/tZa1PgguAp7Z1raVShRS3xd7yGMEFwFO7Tgppc2mR6k08tocF/hyCC4BnnFPSNGvgw+o/60ar/RJcADzT6fHFmluYpDaj1lntl+AC4Ilgahvd33ChdoZqKrR7j9W+CS4AnsjNqCdJysprbr1vgguAJ/a2L5EkLX48zXrfBBcAT6wfOF5dFlyqOi/Zu5r4I4ILgCdCTlhR7yR40rdvPqsIwD+iWjTX+D3bVfc5+6MtiREXAA+svi5J47L7eNY/wQXAunCzQhXsjvWsf4ILgFWF53XT0jOeUZMPg57VILgAWLVxkKMYE6Wa72R5VoPgAmBNsFYt3X7KB5Ikp7TUszpcVQRgTbioSNn5SeqX20XVtMGzOgQXAGucoiKt7CJPQ0tiqgjAhwguAL5jHMez3YYAwBOMuAD4DsEFwHcILgC+w+0QACKWERjq2aL49PCkiHegr9LgSh4z7Zj+0TljB0T8DwFw/GCqCMB3CC4A1gXr1Nb7uVlKWyQF26dY75/gAmBduEVTlTgh3dcwSzlD6lvvn8V5AFZFNWuqFs+u8bQGIy4A1mz8Wy8lvLFP/2o8q+K1mr22a9Nfe6ng/G7W6jDiAmBF9MzGWpHypEqckKSyp58uKg7rltafaEjaD5Kkge+kW6lFcAGw4vLGc1TihMqDS+owY4QazIhRzJ6Q7ugd0LKh47T5jl5q+s/ZrmsRXABcC6a20aAa8/TjSGvK/sZqe+MahfbulSS1WZ2ieYNi9eHIf6l/7G1Kvj9LTlFRpeuxxgXAtXC1n8ZAv9twtl7t26MitCQplL1K178wQo2D1bXw6kdk2rZ0VY/gAmBFtAlqYJN07TrlB5Vuzj3sePLbOxVtgoo2QW25x10tgguAaytHxlWsbR1NzgX1KtbAku5yV481LgCu/eW09456LKpZU+WlJ+np4U9KkuYVxcoUu9sBiOAC4Knsexppef/HJUmT9yXqqVuGKnbFPFd9ElwAPBM9s7H+2Xhyxfcv5PZS7HvuQksiuABY8OCbF+iSqx/R+7lZB611RZvgQd87Zx6+aF8ZLM4DcK3lGzuOemxeUayu2XCWfjNgmLV6BBcA10LZq5T60fVHPHb9cyO085RdCi9ZYa0eU0UAVqRcs0CnX3KDoq/aqo9S31C/W0bLMVLy4u36+Rsljh3BBcCaWq/NkV6Thqib4jVHkqyHlsRUEYAPEVwAfMc4jme7DQGAJxhxAfAdgguA7xBcAHyH4ALgO9zHBSBiGYGhnl3Nmx6eZCJtS3BZljxmWsQ/2JyxAyL+QQH4CVNFAL5DcAHwHYILgDWhPp01cvWaox4PtmllpQ7BBcCaDWfFqG5w31GPb/+3ncghuABYYaKr6cwzF/9sm98mZylYp7brWgQXACvGrpqlbvHrNTa9z1HbjE74ViY+3nUtgguAFU2jSvXKjQMU2rXriMejGjdSwFLkcB8XANd2/r6nXtqzV9GfZh21Tfa9zTQsp59C27a7rseIC4BrgcE7NPHVs496PJjaRv/t+4w2Ppwip6jIdT1GXABciWreTLPSXtWgAV2P2mbHv8LqEhNSjclzrdRkxAXAlfAPu/X37Z0V1bjREY9HNW+mr9Net7a+JTHiAuBSOC9Pn+S2Va/312vWMz0rXt/d3lHN5D3qkZSjsMJWaxJcAFxLuCdWt781Uw/clVnx2oKioEIKqEu1Ykl2nydAcAFwb94yXXHlaO1uHVPxUr3xZSGW+3aqsrq/YLUcwQXAiuDMhao38/DXCzbEK9Dd7nI6wQXAW46sr3FxVRGAp8KxZaG1PeT+/q0fEVwAPPXfs5/WiuKwMibeZq1PpooAPHXv+kHa/2QTnTB5trU+CS4A3uq7WTW02WqXTBUB+I5xHM92GwIATzDiAuA7BBcA3yG4APgOVxUBRCwjMNSzRfHp4UkRfxLbN8HF1vYAfsRUEYDvEFwAfIfgAuCp3Vf21MdbFmvTnb2s9UlwAfBMVJMk/f1vEyRJ2Tc8aa1fgguAZ7ad1Vz940okSZ0XXGStX4ILgCdK+qVr/n1PSZLyw8VqMGiltb4JLgCeWH9BsOLrC1cPtto3wQXAEwO6LpEk7QkXqOTuRKt9E1wAPPF4k7JdqzeXSoEvFlntm+AC4JkVxfm6tf/l1vsluABYVziwmyRpZUlDhVattd4/wQXAuoL6ZQvzt2Vd4En/BBcA64oG79aK4nw1nRDtSf8EFwCrgiknakHX/+rDfR0U/WmWJzUILgBWbfhNoqJNUDPObudZDYILgFUNz8yVJIW27/CsBsEFwBoTE6Pzk8puPHWKijyrQ3ABsCcU0rMrTvW8DMEFwBqntFTJY/ar3ddXeFrHN8+cB+APoTXrdcJQb2sw4gLgO8ZxPNttCAA8wYgLgO8QXAB8h+AC4DsEFwDf4XYIABHLCAz17Gre9PAkE2lbgsuy5DHTIv7B5owdEPEPCsBPmCoC8B2CC4DvEFwArIhq3kxJc+K1+onuCqa2OWq7YIMG2n1lT5mYmMrXqvTfBIAD3DtzstpEh3XmzkYKLV99xDbBBg102VcL1SN2im5Ydp20aHmlajHiAuBaVNMmOqlaUCfNuF4JA44cWpK04r5k/bbmNg1+/DY5lQwtieAC4FJU82Za9a8G6vLQjWo9bOFR2zk9T9aagc+o69gblfTgbFc1CS4Armx6tKZW935BTZ7/+RFUbu8a6jB7mBIfcxdaEsEFwCXHMSpxQgoXFB7xeCA+XqvHdddr1z2sE4Yus1KTxXkAVjScGaONeY1VPLFRxWvfn+bo3O6L9W7Sk5Ls7bHIiAuAK0lDsjU4/VzNWtJWM1Lf1oP/fFLf9Qnruz5htXq1UP9JKpsadpp7pbWajLgAuFb6/ValjNiqc0d0liSlaJ4kKXBSWwVkdN+ODmr+hz0qtVSPERcAz2y8K6iwHH3yj9NVummztX4JLgCe2HFtTy3t8bI2lhao+vZiq30TXAA8kZ+xT5J04eJrFPz86Pd3VQbBBcAT3/R6UXOKpIbnf2u9b4ILgCfCcnT1gmGSpGC9ugq2a22tb4ILgGfCoYC2jeqlzjO2a+M/qlnrl9shAHhmxenPK3y6o9Qvf6dWd+9XyFK/BBcATwzf0FeZc9uq7aNbdOL3KxUqPPJHgiqD4ALgie29dquV5li76fRArHEB8B3jOJ7tNgQAnmDEBcB3CC4AvkNwAfAdrioCiFhGYKhni+LTw5Mi3tmd4EKVSR4zLeJf+pyxAyL+Jcbxh6kiAN8huAD4DsEFwHcILgBWBevX05qXO8nExHhWg+ACYM22Ub10beZcfXvmBAXr1/OsDlcVAVix6rkuWnXWYwrIKCxp6rz3tKqkWBe8fLNa/mORwhafDsGIC4AVn/Z9RJL0wM52umjt2ZKklOhqGn/ZUwo0TrRai+ACYEVyVJxGbDpDX5xUXQXn5Ou1vEQFZHR6rNR+8kZFNW70y51EiOACYEVYjpY+07Hs6/379eb3XRWWo5AT1taieDmFRdZqEVwArBnyx89kunbUjmt76s1W72hFSYkk6eZG0xVq08xaHRbnAVixqqRYt9bL1u1TVygsRxetHaCC0Q005LWZGl5rk9aODujEOXZqMeICYMW1t96k9aWFCpqAAjIqOCdf4cXZGvvZeQrI6IEub1urxYgLgBU1J83VcN2sH36br8I9MUrZv0CS1GZMtvq2vkDTUyfrWUu1CC4A1tScNFc1Jx38WjgvT3undJBSpajGjVT63feu6zBVBOC5Bk9l6txLr9HUBdNk0lNd90dwAagSgS8WKSCjvH8UKBAf764vS+cEAL/o5bxG+rLjWyo9+URX/RBcAKrMm0POkCTtuNXd5xYJLgBVJrRitS5a11/vdZog9Tip0v1wVRFAlco7bYeu1qmSlla6D0ZcAHzHOI5nuw0BgCcYcQHwHYILgO8QXAB8h+AC4DvcDgEgYhmBoZ5dzZsenmQibUtwWZY8ZlrEP9icsQMi/kEB+AlTRQC+w4gLgOeiGiWquHWSJCl6Va5W3tFSdbKN6q4oVGDWomPvz/YJAsCP9lzeQzvPLdSYTh/pylofSJIm7jlBF8RPUcLQWEnSwCbpx9wvwQXAusDJ7TT+vfFqEJyvwCErUlfX3igp1lX/BBcA6/a3iFdisPphrz+9u6Ve2dD1oNdqa80x909wAXAtWKe23ln+mU758yglvJip6lPnaeDUdAXbtVZo5TopHKpoW5mgOhRXFQG4EoiNVdFbtdXm7etVf8ryg46FVqw+KLSs1bTeI4DjRjAhQSsfOlkftZuitveuU2jv3iqpS3ABqLQtl7fTyiFP6t39CQpt315ldQkuAJX27M2PSpJOq/6dQr07V1ldFucBVFq3mGiVOCElBGL17n+fVocZI1R7fqz2NXXUcup+SdKOk2ooceY2hVattVaX4AJQaS3e+71WDXxakhRtglrZb7zUr/zgFT+1mzfG6Kbsi1V34CordZkqAqi0Njcs0lnDrtUreY1V4hz96mG3GEdfdXrFWl1GXAAqzSktVfSnWXqtbZImXDhEoWijXrfM09hG8w9re+gd9G4w4gJgRY235qrWa3O05Ja0iteuyMlQWGHrtQguAFZFL1itHgsvkSS9nDxdAQVU5JQoY/lvrNUguABYFc7LU6MbC5VZFJQkTd5XXydPukkx/XOs1WCNC4B1pTkbNfqR65XXtUBt/7JDrTbMsdo/wQXAE4njZitRUqkHfTNVBOA7BBcA3zGO49luQwDgCUZcAHyH4ALgOwQXAN/hdggAEcsIDPVsUXx6eFLEO7sTXKgyyWOmRfxLnzN2QMS/xDj+MFUE4DsEFwDfIbgA+A7BBcB3CC4Anig+u6tWPd1NJy00apcVpe9v6mWtb4ILgBXhU9O07tU0Pbbha32Qu1C3P/6SEpL2aPLsbro/cbb6XTFHt61dphtWr9LmO9yFGLdDALDiHy+PV3q1oKTqumR9hvLOKlSD/SvVQNKI9H7aNrq5bn97lmYVNNbiUY9p8H/PV+mmzZWqxYgLgCuBGjW09t89lF4tqPlFjlLevF77BpYqvH9/RZuO8bkKRwVUL1Bdg2vsVkDubtNjxAXAld2DOuqzoQ9pRkGCxl4/TK0+maMfNyozUVEKtDlRE6bW1YMvvShJCpqAOs69VE22VX6DWIILgCt13l+uK4dfrumpk9X/hQkKOYfu6jOv4qtTlw5V3RtCSlqXLTefHSK4ALgSzstTTP88pTw1Uv3Tl2nVnobakFtfQzou0r8aLTiobcJVeSrdus11Tda4AFiRMnKecroVqFrGBrW+KkufbGxbcSynNF9p/xmlkIXQkhhxAfBIZtfnJFWTJF34r9uU9MRsa30z4gJg3ZZbe6m6KQutR3e1UqPnF1vtn+ACYNXqcd21+KbH9V0oX71HjdTHHWopnJ9vtQbBBcCaYL26WnTBI5Kk3l+NUtyUuZ7UIbgAWBFMSNBNc2eppomRJLX+/WrPahFcAKzYMait+seVVHx/4J3zthFcAKz4zS2fVtx82uq9EZ7WIrgAWPH82xllH+d5apRSRsz75b/gAvdxAbCi+V2zddZdaWome/drHY1xHM92GwIATzBVBOA7BBcA3yG4APgOwQXAd7iqCCBiGYGhnl3Nmx6eFPHznKs0uJLHTDumf3TO2AHuHkwN4P8lpooArCo8r5tMVJTW399TTefU1Pr7e2r9/T1VeF43azWYKgKwIli/nkJvVNfrrR/W1lC0OlYrv3t+2JeSpG2X5+u8Rreq3vhM17UILgBWrHr0BK1sO1FSnBoGpWs2naHN++tIkoImrGlt3tMbf3lQI1aMUuArdw8WZKoIwIo3ej0jSfqoIE69R43U1nOiFOi7SYG+m+T0+14pb16vE6NrquAvexXVKNFVLYILgBVp1aLUYfYwjWvVVnFT5iq0a9dPB8MhtfrjHL2el6AvOr6lteMauqpFcAGwJpgV/7PH75w/WJJ0Q+qXruoQXABcC5zUVqtKClV/acnPtkv4ItZOPSu9ADiurR5WR79bcYViPphfJfUILgCu/fGcaap1ztoqq0dwAXAlWKe24gMFVVqT4ALgyuarU3VZ/LaI2hadu0eSlB+u5qomwQWgyrzeaaIkacoDfV31Q3ABqBKlZ6arbXSMrs89RXXeWOiqL4ILgCuN/z1bT+9prqimTQ47Fj41TWtfTVPaIumTl8erzw0jlNOtQE5JsauafFYRgBWd3t+oBT+ccNBrY1s8q7RqZTGTVRxSzc++VchCLYILgGsvPDRQc/7+hNRgySFHolSqkJYUS5e/MVot9rp/MkRZrwDgUt3nMvXwza11c8Lqg15v+8XvVG1ZnJr+c7ZayE5oSQQXAEs+7RCvT9X5oNdayt3ja46GxXkAvkNwAfAd4ziebdoBAJ5gxAXAdwguAL5DcAHwHYILgO9wHxeAiGUEhnp2NW96eFLEO9cTXD6XPGZaxL9IOWMHRPyLAfyaMVUEYE0gLk5dF4f0fm6Wt3U87R3AcSMQF6dVz7bRXxssVFhhT2sxVQRgxbo7T1Z2n3G6bN052vmPFqom73b8YcQFwIrihqWSpKWzWqvaR95uU0ZwAbAiumax8sLFOmF6kee1mCoCsGL56c/pD1v6Kvj5kZ8nX3ROV+U1i1KDrL1yspa7qkVwAbDm61c7q5FmH/Ta2lc66dHur6ljta+UGIzRmpJSnf/WH3XiLXMqXYepIgDXgrVq6ZOCGmr0n59Cq6R/F922dplW9J6gAXGF+j4Uo4/zayu1WnU9OXiigu1TKl2P4ALg2vqbOmj0nEsqvg+2aqEPn39Kp8YWakZBnFpOuU53n3WxnmidoowV5+mM6vkqOKF2pesxVQTgWrt+q5X9eWtJkklP1djJE5U6c6RaX71C4cJCtdbcit191qxrpL/V7arYmcsqfbcXIy4ArjWN2y2Zsk+frRodo3bR0TrxskUKFxYe1ja6ZrH2l8Yc8VikCC4AroWdgOSUfRS2ceLuo945H2zVQstPf05ZO5q6qkdwAbCqztWFmlsUrXUP9FRUk6SDjrV7Y4O2hgoU+2hdVzUILgCuBFu10Om1v634vjR3ix7oN1jfXD5OjSbvVbBeWUjtvqKn/pb4tXq/eYvrO+tZnAfgSmjNer0wsJ+Wff6Y/nDuKZr3dCfV3FKqdpNH6dyeizVl6ecKKKBn9+zUb5v21Imq/P1bPyK4ALgWWrVWz+9tpv8kzZLunXXY8Q5fDVerm3dI2mKlHsEFwIqpvTto3PDB2t+iRB+f/YjO+ugmSVKbCYVKnr9UpRZrEVwArAht3aYmY7dJkm7UKUopf6yNF896ZnEegO8QXAB8h+AC4DvGcTzbbQgAPMGIC4DvEFwAfIfgAuA7BBcA3+EGVAARywgM9exq3vTwJBNpW4ILVSZ5zLSIf+lzxg6I+JcYxx+migB8h+AC4DsEFwDfIbgAWBfq3Vnv52bp4y2L9X5ulnZNa61AWnvtvqKnAvHxrvtncR6ANcVnddFHzz0lab5OuXO0ovPLrscUNA/o+1Ok6WMe1OCCm1Xjrbmu6hBcAKxwep6sfz71jCTpkrXnKuGFzIpjtVu1UNqkter63h+V4jK0JKaKACzZdWeB0mOkgd9eoPw/NDzo2O70RN3VMEvNPrFTi+ACYMXsTq9pc2mBAncmyFm0vOJ1ExOjVjdlK6CAqk+dZ6UWU0UAVoQV1obSWtKcpZKkYGobrbixtr497wlJ0oyCmtZqEVwArOkUs1+nLS2UJHWLe1t9qhdW7Gn9pyUXqqmWH/0vHwOmigCsGPTtEMWZarq13jLdWm+Zzqier9Nuv0GnLb5UktRwfJy1Woy4ANjRd7POHHK9tqWXjYeS/5Kp7S8X6du01zVxT7Liln9nbYsygguANXFT5ip5yk/ff3vmBIUV1hMrz1DSpmxrdZgqAvDEhjc7SpK6zb9SSUPshZZEcAHwyLvdn5YkmRkJ1vsmuAB4okVUrMIKK6rQ/rMHCS4AnggrrEd+aK964zN/ufExIrgAeOa5d/p50i/BBcATf/6+uxIXhDzpm9shAHjim/SwqsvOZxMPZRzHs007AMATTBUB+A7BBcB3CC4AvkNwAfAdrioCiFhGYKhnV/OmhydFvHt5lQbXsWzBLrENO4AjY6oIwHcILgC+Q3ABsC6qUaJWPd3toJ2sVz3dzV7/1noCcNwzMTFad09nPX7hBJ1RPV8lTkBhhTUr7VUpTRo0oquVOoy4AFiz8dZ0LbtinM6oni9JGr6hryd1CC4AVoRnNNOSkY9Jkqbsa6hBzXpoe6/dOr/lqTpn+EgtKAqq3zd5VmoRXACs+KDtVEWboAqdUt315sVSuOyRNk5RkaI/WaDLPx6hW+uu1frXTnZdi+ACYEVYYZU4IQ1fN0jJfz38qacpI+epxAnpsvbzXdciuAC4FtUkqeLrdZNae16P4ALg2qrRzSVJfW8YqcTHZv9s2zH1l2jfb3u4qkdwAXAtFBdWQAFVn/rzTzyNNkEFLMQOwQXAtZNOylFY4V9sV+KEImr3SwguAFVqQ2mxqm8vdtUHwQWgSp3//K0Kfr7QVR8EFwDXCs7YqoACmrjxK4X6dD5im/Wvn6Reiy7RCXf//OJ9JAguAFZ0mX+5EoPVtbl37GHH9v+mu97sPl517zv8WGUQXACsaHJnWFP219W7Vz2o1U90lySZ9FRtG9VLDz/0uNpVC0hzllqpxdMhAFgRWr5SL57dR888G9ZHAx/WBZtu1YRrH1OnmLKriGdnX6hq2mClFsEFwJrSdTmKuaSBRnT6g7Kef1Rt37tBktTi7bBiPl8qWw+sJ7gAWBXavl3Rn2zXoCZdlaKfbki1ucsGa1wAfMc4jme7DQGAJxhxAfAdgguA7xBcAHyHq4oAIpYRGOrZovj08KSId65nxAXAd3wz4koeMy3ipM8ZOyDi5AbgP4y4AHji4y2LlZhZy5O+CS4Annmp+ZfKH9Lder8EFwBPbTnd/soNwQXAU63+OMd6nwQXAE+t+Y+7rciOhOAC4DsEFwDfIbgA+A7BBcBTLM4DgAguAD5EcAHwHYILgKe4jwuAL3jx+cQDEVwArGtx2wpP+ye4AFiVP6S7Xmr+pSTptBuu8+R2CN88SBCAv5yVlKY4zfWkb0ZcAKyKmzJXZyWleVqD4ALgOwQXAN8xjuPZbkMA4AlGXAB8h+AC4DsEFwDfIbgA+I4nN6DWr1/fSU5O9qJrQFlZWTscx2nwvz6P41FGYKhnV/OmhydFvI+ZJ8GVnJysBQsWeNF1lUkeMy3itjljB3h4JjiUMWbD//qFn35tAAAFwUlEQVQc8L/FVBGA7xBcAKwrPTNdg7J36v3cLH2Qu1Dv52YpfVFY6tbRSv98yBqAFasmdNEl6fMkSfc0fFaSFFZY7WZeq4bvxij+jTmSllmpRXABsGLVOc9oa6hAT+7spZQPr1O9zGjVm5ipE7XIei2CC4AVYYX15M5eyuoUUIq8vTjHGhcAKzr8d7TuabhIoT6dPa/FiAuAHUYKyGhnaqzqmnRJUsyC1Qrt3Wu9FMEFwIqxg19RWI7m3PGoAuWTud7LhqpoUqrqTcy0WovgAuBaVLOmGlQjS3dt66T3cjrImVNHgy7+Sje3/FSD792t8L2Ozr7iWkV9lmWnnpVeABzXSjdt1sAmZdPDJGVLkrIeCChLLfVseZutt8fouYnzdcfVI1wHGIvzAKpEkwdma25+K907Ybx2Xt3TVV8EF4AqM+nPZ2nanjQ9+Zdx2nh3r0r3Q3ABqDLV35mnJec109z8Vlr8+0cr3Q/BBaBKlW7O1bglfSquPFYGwQXAiq03/vLUL9g+Rfs/aqnPT31c5w2+qtK1uKoIwLWdV/fUvh4FSnzs8GNRzZpqw6UnSJImXPeY5hScqCF336q68yt/bxfBBcCKFb0naNH6sC7N/L2MpMS6e/V5x0kKaKHCchSQUasPr1P7u79T3c3ubkgluAC4Vm9iptqdeY0k6cWeE9UtxtG1m3qr3cxrFN5ZTS2nlEiSUj5foFIL9QguAFaceFnZ42vu1Y8fss7z5JE2EovzAHyI4ALgO8Zx7O82ZIzZLulYdmKpL2mH9ROpnF/TuUi/rvP5tZxLc7YnO755ElzHfBLGLHAcp8v/+jykX9e5SL+u8/k1nQuOb0wVAfgOwQXAd34twfXsLzepMr+mc5F+XefzazoXHMd+FWtcAHAsfi0jLgCIGMEFwHeqNLiMMWcbY1YaY9YYY8Yc4XiMMeaN8uNzjTHJHp1HM2PM58aYbGPMcmPMH47QprcxZo8xZnH5n795cS4H1Msxxiwrr3XYbpqmzLjy92apMcaTzeuMMW0O+DcvNsbsNcbcdEibKn1vgENV2WcVjTFBSU9IypC0WdJ8Y8y7juNkH9Dsakm7HMdpZYy5WNIDki7y4HRKJf3JcZyFxph4SVnGmOmHnIskzXIcZ6AH9Y+mj+M4R7vB8xxJrcv/dJf0VPl/rXIcZ6WkNKniZ5YracoRmlb1ewNUqMoRVzdJaxzHWec4TrGk1yWdf0ib8yW9WP71W5L6GmOM7RNxHOc7x3EWln+dJ2mFpCa261h2vqSXnDJzJNUxxjT2uGZfSWsdxzmWT0EAnqvK4GoiadMB32/W4WFR0cZxnFJJeyTV8/KkyqejnSTNPcLhnsaYJcaYD40xqV6ehyRH0ifGmCxjzLVHOB7J+2fbxZJeO8qxqnxvgIMc14+1McbUlDRZ0k2O4xy6T/hClX0mbp8x5lxJU1U2TfPKqY7j5BpjGkqaboz51nGcLz2s97OMMdUkDZJ0xxEOV/V7AxykKkdcuZKaHfB90/LXjtjGGBMlqbaknV6cjDEmWmWh9YrjOG8fetxxnL2O4+wr//oDSdHGmPpenEt5jdzy/25T2ZpSt0OaRPL+2XSOpIWO42w99EBVvzfAoaoyuOZLam2MaVH+f/OLJb17SJt3JQ0r//pCSZ85HtwhW75uNlHSCsdxHj5Km0Y/rq8ZY7qp7L3yKkRrlF8kkDGmhqT+kr45pNm7kq4sv7rYQ9Iex3G+8+J8yl2io0wTq/K9AY6kyqaKjuOUGmNGSfpYUlDSc47jLDfG3CtpgeM476osTF42xqyR9IPKws0Lp0i6QtIyY8zi8tf+LOmE8nN9WmXBOdIYUyqpQNLFXoRouURJU8qzIErSq47jfGSMGXHA+Xwg6VxJayTlSxru0bn8GJ4Zkq474LUDz6Uq3xvgMHzkB4DvcOc8AN8huAD4DsEFwHcILgC+Q3AB8B2CC4DvEFwAfOf/AIJiZlYe4/WeAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x576 with 32 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Replicate our model function onto 8 cores (with no inputs)\n",
    "fit_with_infeed = tf.contrib.tpu.batch_parallel(\n",
    "    fit_batch_with_infeed, num_shards=8)\n",
    "\n",
    "\n",
    "# Infeed requires a static shape for inputs.  We create a new set of placeholders\n",
    "# here with a fixed batch size.\n",
    "image_batch = tf.placeholder(name='images', dtype=tf.float32, shape=[BATCH_SIZE, 28, 28])\n",
    "label_batch = tf.placeholder(name='labels', dtype=tf.int32, shape=[BATCH_SIZE,])\n",
    "\n",
    "session.run(tf.global_variables_initializer())\n",
    "infeed_ops, outfeed_ops = setup_feed(image_batch, label_batch)\n",
    "\n",
    "# # Start training.  We first push a batch of data on the infeed so the device\n",
    "# # is working while we're getting the next batch ready.\n",
    "print('priming infeed')\n",
    "session.run(tf.group(infeed_ops), {image_batch: x_train[:BATCH_SIZE], \n",
    "                                   label_batch: y_train[:BATCH_SIZE]})\n",
    "\n",
    "for i in range(50):\n",
    "  _, _, loss = session.run(\n",
    "      [tf.group(infeed_ops), fit_with_infeed, tf.tuple(outfeed_ops)], \n",
    "      {image_batch: x_train[:BATCH_SIZE], \n",
    "       label_batch: y_train[:BATCH_SIZE]}\n",
    "  )\n",
    "  if i % 10 == 0:\n",
    "    print('loss = %f' % np.mean(loss))\n",
    "\n",
    "_, final_loss = session.run([fit_with_infeed, tf.tuple(outfeed_ops)])\n",
    "print('final loss = %f' % np.mean(final_loss))\n",
    "\n",
    "[predictions] = session.run(predict_on_tpu, {\n",
    "    images: x_test[:16], labels: y_train[:16]\n",
    "})\n",
    "\n",
    "plot_predictions(x_test[:16], predictions)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "KQac9EiYZb0I"
   },
   "source": [
    "## Training Loops\n",
    "\n",
    "In our previous example, we used infeed and outfeed to decouple the CPU and TPU operations, but our TPU is still dependent on the CPU to \"pump\" the fit operation once per training loop. What if we could put a loop around the entire TPU computation such that it was entirely decoupled from the CPU? The tpu.repeat function helps us do just that. We supply a function to run in the loop and the number of times we want to run:\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "colab": {
     "height": 872
    },
    "colab_type": "code",
    "id": "hK13OKSbg20x",
    "outputId": "c2d89a4d-6df9-4ba8-c41c-a95d984058d4"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tensor(\"infeed_dequeue_2:0\", shape=(128, 28, 28), dtype=float32) Tensor(\"tpu_5/add:0\", shape=(128, 10), dtype=float32) Tensor(\"infeed_dequeue_2:1\", shape=(128,), dtype=int32)\n",
      "Infeed 0\n",
      "Infeed 10\n",
      "Outfeed: 0 [[182.41867], [166.20851], [174.40918], [166.83932], [172.24617], [209.3718], [196.6448], [192.4896]]\n",
      "Outfeed: 10 [[211.57326], [250.54344], [109.76998], [157.62318], [151.5726], [111.26413], [170.51817], [112.66968]]\n",
      "Infeed 20\n",
      "Outfeed: 20 [[80.74988], [191.61298], [149.25012], [35.894707], [92.90561], [125.87827], [150.47806], [58.80836]]\n",
      "Infeed 30\n",
      "Outfeed: 30 [[51.099712], [99.81317], [51.137062], [9.362455], [33.58825], [12.327646], [52.579407], [30.45329]]\n",
      "Infeed 40\n",
      "Outfeed: 40 [[33.157497], [63.580967], [8.313176], [8.708943], [13.578816], [0.20352554], [17.062672], [2.2809372]]\n",
      "Infeed 50\n",
      "Outfeed: 50 [[3.0567093], [24.351074], [0.3786888], [3.2669644], [0.0], [1.6983624], [2.5881014], [0.0]]\n",
      "Infeed 60\n",
      "Outfeed: 60 [[0.0], [12.785406], [0.0], [0.48258018], [0.0], [0.0], [0.0], [0.0]]\n",
      "Infeed 70\n",
      "Outfeed: 70 [[5.587593e-09], [2.7874959], [0.0710191], [0.0], [0.0], [0.0], [0.0], [0.0]]\n",
      "Infeed 80\n",
      "Outfeed: 80 [[0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [2.8463585e-06], [0.0]]\n",
      "Infeed 90\n",
      "Outfeed: 90 [[0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0], [0.0]]\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAS4AAAHVCAYAAABPFRaZAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzt3Xd4FVX+x/HPuTchIRAgtEAACQihBDQQug2EgAoiuLJ20dVVVGRd14Lr7lp+rour6yp2AetaEcGCDVEUJbRQJUgPJShNSiD93vn9kRipesmcyTLL+/U8PCZ3Duc73OT5eM6ZuXOM4zgCAD8J/LdPAACOFsEFwHcILgC+Q3AB8B2CC4DvEFwAfIfgAuA7BBcA3yG4APhO1H/7BAD4R0ZgmGcftZkWnmgibUtwHaeSR089ql/AnDEDI/6lArzGVBGA7xBcAHyH4ALgOwQXAN8huAC4suP3PVXjqwZa82onFQ7qpkBae89rclURgCuxO8PKntFKUZJ2XrNLd3Z4RxnVCw5oU+AUa05RDfWOLVGrqdcp5dp5rmoSXABcqfH2HNV4++fvH2/UW/efkiztdwNNVEFYNZZ8r96z31NcTrTrmgQXAKtKf9iiGpO2HPL6lmt66uEf2yj5hbUqdVmDNS4Anotq3kxP/PkJTXysn0q//8F1fwQXAE8VDeyq1755S11jjOqNz7TSJ8EFwFMbzg6oponRxesyrPVJcAHwTCA+Xpef9rX2hAu19YGW1vplcR6AZ1bdk6oP6j+l81b9RjEfursFYn+MuAB4YvdlPbTkwrFaU1qgvQ82tdo3Iy4A1pmoKH3z4FN6NS9Jr1+YoZjF9kZbEiMuAF44uY0k6ckHhim8eLn17gkuAFYF26doxBvvqv3zN6rOK7M9qUFwAbDquxsSdG7cHjWdUSw53jzpmeACYE3hud00/dx/eV6H4AJgzeZTgjohKk7/2NFeUdOzPKvDVUUA1rQcnalzRnf2vI5xPJqDAoBXmCoC8B2CC4DvEFwAfIfFeQARywgM82xRfFp4YsS7pf/PB9fRbDXPNvOAP/zPBxeAqpNzf0+FYh01SN2mzJMn6cTPr1L83OpKHDvLah2CC4AVO6e21rdpT1R8X+JI3/UZr1e7NNZb085QaPkqa7VYnAdgxTdpb1R8/cyuluqffb4k6dL477XqyvpWaxFcAFwr7ZsuSXp0Z4rO63+JPjq1pWLP3aIHtncsO57gdkOyAxFcAFzb26SaHt2ZohmDOyq0bIVCO3dq9b2dNKrufElS04/tRg3BBcC1Oi9n6rMO8Spdm1PxWuYlD6tmIEZ/+r6H4j+z+zBBgguAdbsu76nagVhlFgW16P5OCu3ZY7V/gguAdds7l90+OXzGNao+Za71/rkdAoBVxdOaK7Ptv3Ry5nVq96c1CnlQg+ACYE2b+dH6V+N3JMWq2QXfehJaElNFAJbsHN5T9yZ+KUkantPP01oEFwArThs1RzUDMZKkzOxWntYiuABYMaZR2aavfZYOU7vbV3tai+ACYFXtG8IK7dzpaQ0W5wFYNWXmpAO+77XwYm3fUksJDfI0J/21Q9q3f3WkWt6eeVQ1CC4AnprV6fWKr/OdYpU4YUnSOUuu1O5F9dXk66P/HCPBBcCK1JdGytkvUbIv/fkRN6kzr5KzoYZavr1XmrtUkpSgVUpQ5R51Q3ABsKLFnw+c7g26Pf3nY1pitRaL8wB8h+AC4DsEFwDfMY7j2W5DAOAJRlwAfIfgAuA7BBcA3yG4APgON6ACiFhGYJhnV/OmhSeaSNv6JriSR0+N+A3LGTMw4jcAgP8wVQTgOwQXAE+Z9FR9mLtAG//ay1qfBBcAT23tWkulCilus73lMYILgKd2nhTSptIi1ZtwdA8L/CUEFwDPOKekaeagR9R/5k1W+yW4AHim0xOLNKcwSW1GrrXaL8EFwBPB1DZ6oOEC7QjVVGjXbqt9E1wAPJGbUU+SlJXX3HrfBBcAT+xpXyJJWvREmvW+CS4Anlg3aJy6zL9EdV62dzXxJwQXAE+EnLCi3k3wpG/ffFYRgH9EtWiucbu3qe7z9kdbEiMuAB5YdV2Sxmb38ax/gguAdeFmhSrYFetZ/wQXAKsKz+2mJWc8qyYfBT2rQXABsGrDYEcxJko1383yrAbBBcCaYK1auuOUDyVJTmmpZ3W4qgjAmnBRkbLzk9Qvt4uqab1ndQguANY4RUVa0UWehpbEVBGADxFcAHzHOI5nuw0BgCcYcQHwHYILgO8QXAB8h9shAEQsIzDMs0XxaeGJEe9AX6XBlTx66lH9o3PGDIz4HwLg+MFUEYDvEFwArAvWqa0PcrOUtlAKtk+x3j/BBcC6cIumKnFCur9hlnKG1rfeP4vzAKyKatZULZ5b7WkNRlwArNnwt15KeHOv/tl4ZsVrNXtt08a/9lLBed2s1WHEBcCK6BmNtTzlKZU4IUllTz9dWBzWra0/1dC0HyVJg95Nt1KL4AJgxWWNZ6vECZUHl9Rh+gg1mB6jmN0h3dk7oKXDxmrTnb3U9B+zXNciuAC4Fkxto8E15uqnkdbkfY3V9qbVCu3ZI0lqsypFcwfH6qPr/6n+sbcr+YEsOUVFla7HGhcA18LVfh4D/W79WXqtb4+K0JKkUPZK3fDiCDUOVteCqx+VadvSVT2CC4AV0SaoQU3StfOUH1W6KfeQ48nv7FC0CSraBLX5Xne1CC4Arq24Pq5ibetIcs6vV7EGlnS3u3qscQFw7S+nvX/EY1HNmiovPUnPXPWUJGluUaxMsbsdgAguAJ7KvreRlvV/QpI0aW+inr51mGKXz3XVJ8EFwDPRMxrrH40nVXz/Ym4vxb7vLrQkgguABQ+9db4uvvpRfZCbdcBaV7QJHvC9c+ahi/aVweI8ANdavrn9iMfmFsXqmvUD9JuBw63VI7gAuBbKXqnUj2847LEbnh+hHafsVHjxcmv1mCoCsCLlmvk6/eIbFX3lFn2c+qb63TpKjpGSF23TL98ocfQILgDW1Hp9tvS6NFTdFK/ZkmQ9tCSmigB8iOAC4DvGcTzbbQgAPMGIC4DvEFwAfIfgAuA7BBcA3+E+LgARywgM8+xq3rTwRBNpW4LLsuTRUyP+weaMGRjxDwrAz5gqAvAdgguA7xBcAKwJ9ems61etPuLxYJtWVuoQXACsWT8gRnWDe494fNu/7EQOwQXAChNdTWeeuegX2/w2OUvBOrVd1yK4AFgxZuVMdYtfpzHpfY7YZlTCdzLx8a5rEVwArGgaVapXbxqo0M6dhz0e1biRApYih/u4ALi24/c99fLuPYr+LOuIbbLva6bhOf0U2rrNdT1GXABcCwzZrgmvnXXE48HUNvpP32e14ZEUOUVFrusx4gLgSlTzZpqZ9poGD+x6xDbb/xlWl5iQakyaY6UmIy4AroR/3KX/29ZZUY0bHfZ4VPNm+ibtDWvrWxIjLgAuhfPy9GluW/X6YJ1mPtuz4vVd7R3VTN6tHkk5CitstSbBBcC1hHtjdcfbM/Tg3ZkVr80vCiqkgLpUK5Zk93kCBBcA9+Yu1eVXjNKu1jEVL9UbVxZiue+kKqv7i1bLEVwArAjOWKB6Mw59vWB9vALd7S6nE1wAvOXI+hoXVxUBeCocWxZa20Lu79/6CcEFwFP/OesZLS8OK2PC7db6ZKoIwFP3rRusfU810QmTZlnrk+AC4K2+m1RDm6x2yVQRgO8Yx/FstyEA8AQjLgC+Q3AB8B2CC4DvcFURQMQyAsM8WxSfFp4Y8SexfRNcbG0P4CdMFQH4DsEFwHcILgCe2nVFT32yeZE23tXLWp8EFwDPRDVJ0v/9bbwkKfvGp6z1S3AB8MzWAc3VP65EktR5/oXW+iW4AHiipF+65t3/tCQpP1ysBoNXWOub4ALgiXXnByu+vmDVEKt9E1wAPDGw62JJ0u5wgUruSbTaN8EFwBNPNCnbtXpTqRT4cqHVvgkuAJ5ZXpyv2/pfZr1fgguAdYWDukmSVpQ0VGjlGuv9E1wArCuoX7Ywf3vW+Z70T3ABsK5oyC4tL85X0/HRnvRPcAGwKphyouZ3/Y8+2ttB0Z9leVKD4AJg1frfJCraBDX9rHae1SC4AFjV8MxcSVJo23bPahBcAKwxMTE6L6nsxlOnqMizOgQXAHtCIT23/FTPyxBcAKxxSkuVPHqf2n1zuad1fPPMeQD+EFq9TicM87YGIy4AvmMcx7PdhgDAE4y4APgOwQXAdwguAL5DcAHwHW6HABCxjMAwz67mTQtPNJG2JbgsSx49NeIfbM6YgRH/oAD8jKkiAN8huAD4DsEFwIqo5s2UNDteq57srmBqmyO2CzZooF1X9JSJial8rUr/TQDYz30zJqlNdFhn7mik0LJVh20TbNBAl369QD1iJ+vGpddJC5dVqhYjLgCuRTVtopOqBXXS9BuUMPDwoSVJy+9P1m9rbtWQJ26XU8nQkgguAC5FNW+mlf9soC4P36TWwxccsZ3T82StHvSsuo65SUkPzXJVk+AC4MrGx2pqVe8X1eSFXx5B5fauoQ6zhivxcXehJRFcAFxyHKMSJ6RwQeFhjwfi47VqbHe9ft0jOmHYUis1WZwHYEXDGTHakNdYxRMaVbz2w2mOzum+SO8lPSXJ3h6LjLgAuJI0NFtD0s/RzMVtNT31HT30j6f0fZ+wvu8TVqvXCvXvpLKpYac5V1iryYgLgGulP2xRyogtOmdEZ0lSiuZKkgIntVVARvdv76Dmf9itUkv1GHEB8MyGu4MKy9Gnfz9dpRs3WeuX4ALgie3X9tSSHq9oQ2mBqm8rtto3wQXAE/kZeyVJFyy6RsEvjnx/V2UQXAA88W2vlzS7SGp43nfW+ya4AHgiLEdXzx8uSQrWq6tgu9bW+ia4AHgmHApo68he6jx9mzb8vZq1frkdAoBnlp/+gsKnO0r96ndqdc8+hSz1S3AB8MRV6/sqc05btX1ss078YYVChYf/SFBlEFwAPLGt1y610mxrN53ujzUuAL5jHMez3YYAwBOMuAD4DsEFwHcILgC+w1VFABHLCAzzbFF8WnhixDu7E1yoMsmjp0b8S58zZmDEv8Q4/jBVBOA7BBcA3yG4APgOwQXAqmD9elr9SieZmBjPahBcAKzZOrKXrs2co+/OHK9g/Xqe1eGqIgArVj7fRSsHPK6AjMKSpsx9XytLinX+K7eo5d8XKmzx6RCMuABY8VnfRyVJD+5opwvXnCVJSomupnGXPq1A40SrtQguAFYkR8VpxMYz9OVJ1VVwdr5ez0tUQEanx0rtJ21QVONGv95JhAguAFaE5WjJsx3Lvt63T2/90FVhOQo5YW0pipdTWGStFsEFwJqhf/xcpmtHbb+2p95q9a6Wl5RIkm5pNE2hNs2s1WFxHoAVK0uKdVu9bN0xZbnCcnThmoEqGNVAQ1+foatqbdSaUQGdONtOLUZcAKy49rabta60UEETUEBGBWfnK7woW2M+P1cBGT3Y5R1rtRhxAbCi5sQ5ukq36Mff5qtwd4xS9s2XJLUZna2+rc/XtNRJes5SLYILgDU1J85RzYkHvhbOy9OeyR2kVCmqcSOVfv+D6zpMFQF4rsHTmTrnkms0Zf5UmfRU1/0RXACqRODLhQrIKO/vBQrEx7vry9I5AcCveiWvkb7q+LZKTz7RVT8EF4Aq89bQMyRJ229z97lFggtAlQktX6UL1/bX+53GSz1OqnQ/XFUEUKXyTtuuq3WqpCWV7oMRFwDfMY7j2W5DAOAJRlwAfIfgAuA7BBcA3yG4APgOt0MAiFhGYJhnV/OmhSeaSNsSXJYlj54a8Q82Z8zAiH9QAH7GVBGA7zDiAuC5qEaJKm6dJEmKXpmrFXe2VJ1so7rLCxWYufDo+7N9ggDwk92X9dCOcwo1utPHuqLWh5KkCbtP0Pnxk5UwLFaSNKhJ+lH3S3ABsC5wcjuNe3+cGgTnKXDQitTVtTdIinXVP8EFwLp9LeKVGKx+yOvP7GqpV9d3PeC12lp91P0TXABcC9aprXeXfa5T/jxSCS9lqvqUuRo0JV3Bdq0VWrFWCocq2lYmqA7GVUUArgRiY1X0dm21eecG1Z+87IBjoeWrDggtazWt9wjguBFMSNCKh0/Wx+0mq+19axXas6dK6hJcACpt82XttGLoU3pvX4JC27ZVWV2CC0ClPXfLY5Kk06p/r1DvzlVWl8V5AJXWLSZaJU5ICYFYvfefZ9Rh+gjVnhervU0dtZyyT5K0/aQaSpyxVaGVa6zVJbgAVFqL93+vlYOekSRFm6BW9Bsn9Ss/ePnP7eaONro5+yLVHbTSSl2migAqrc2NCzVg+LV6Na+xSpwjXz3sFuPo606vWqvLiAtApTmlpYr+LEuvt03S+AuGKhRt1OvWuRrTaN4hbQ++g94NRlwArKjx9hzVen22Ft+aVvHa5TkZCitsvRbBBcCq6Pmr1GPBxZKkV5KnKaCAipwSZSz7jbUaBBcAq8J5eWp0U6Eyi4KSpEl76+vkiTcrpn+OtRqscQGwrjRng0Y9eoPyuhao7V+2q9X62Vb7J7gAeCJx7CwlSir1oG+migB8h+AC4DvGcTzbbQgAPMGIC4DvEFwAfIfgAuA73A4BIGIZgWGeLYpPC0+MeGd3gus4lTx66lH9AuaMGRjxLxXgNaaKAHyH4ALgOwQXAN8huAD4DsEFwBPFZ3XVyme66aQFRu2yovTDzb2s9U1wAbAifGqa1r6WpsfXf6MPcxfojideVkLSbk2a1U0PJM5Sv8tn6/Y1S3XjqpXadKe7EON2CABW/P2VcUqvFpRUXRevy1DegEI12LdCDSSNSO+nraOa6453ZmpmQWMtGvm4hvznPJVu3FSpWoy4ALgSqFFDa/7VQ+nVgppX5CjlrRu0d1Cpwvv2VbTpGJ+rcFRA9QLVNaTGLgXk7rZARlwAXNk1uKM+H/awphckaMwNw9Xq09n6aaMyExWlQJsTNX5KXT308kuSpKAJqOOcS9Rka+U3iCW4ALhS54NluuKqyzQtdZL6vzheIefgXX3mVnx16pJhqntjSElrs+Xms0MEFwBXwnl5iumfp5Snr1f/9KVaubuh1ufW19COC/XPRvMPaJtwZZ5Kt2x1XZM1LgBWpFw/VzndClQtY71aX5mlTze0rTiWU5qvtH+PVMhCaEmMuAB4JLPr85KqSZIu+OftSnpylrW+GXEBsG7zbb1U3ZSF1mM7W6nRC4us9k9wAbBq1djuWnTzE/o+lK/eI6/XJx1qKZyfb7UGwQXAmmC9ulp4/qOSpN5fj1Tc5Dme1CG4AFgRTEjQzXNmqqaJkSS1/v0qz2oRXACs2D64rfrHlVR8v/+d87YRXACs+M2tn1XcfNrq/RGe1iK4AFjxwjsZZR/neXqkUkbM/fW/4AL3cQGwovndszTg7jQ1k737tY7EOI5nuw0BgCeYKgLwHYILgO8QXAB8h+AC4DtcVQQQsYzAMM+u5k0LT4z4ec5VGlzJo6ce1T86Z8xAdw+mBvA/iakiAKsKz+0mExWldQ/0VNPZNbXugZ5a90BPFZ7bzVoNpooArAjWr6fQm9X1RutHtCUUrY7Vyu+eH/6VJGnrZfk6t9Ftqjcu03UtgguAFSsfO0Er2k6QFKeGQemajWdo0746kqSgCWtqm/f15l8e0ojlIxX42t2DBZkqArDizV7PSpI+LohT75HXa8vZUQr03ahA341y+v2glLdu0InRNVXwlz2KapToqhbBBcCKtGpR6jBruMa2aqu4yXMU2rnz54PhkFr9cbbeyEvQlx3f1pqxDV3VIrgAWBPMiv/F43fNGyJJujH1K1d1CC4ArgVOaquVJYWqv6TkF9slfBlrp56VXgAc11YNr6PfLb9cMR/Oq5J6BBcA1/549lTVOntNldUjuAC4EqxTW/GBgiqtSXABcGXT1am6NH5rRG2LztktScoPV3NVk+ACUGXe6DRBkjT5wb6u+iG4AFSJ0jPT1TY6RjfknqI6by5w1RfBBcCVxv+apWd2N1dU0yaHHAufmqY1r6UpbaH06Svj1OfGEcrpViCnpNhVTT6rCMCKTh9s0PwfTzjgtTEtnlNatbKYySoOqebn3ylkoRbBBcC1Fx8epNn/96TUYPFBR6JUqpAWF0uXvTlKLfa4fzJEWa8A4FLd5zP1yC2tdUvCqgNeb/vl71RtaZya/mOWWshOaEkEFwBLPusQr8/U+YDXWsrd42uOhMV5AL5DcAHwHeM4nm3aAQCeYMQFwHcILgC+Q3AB8B2CC4DvcB8XgIhlBIZ5djVvWnhixDvXE1w+lzx6asS/SDljBkb8iwEcy5gqArAmEBenrotC+iA3y9s6nvYO4LgRiIvTyufa6K8NFiissKe1mCoCsGLtXScru89YXbr2bO34ewtVk3c7/jDiAmBFccNSSdKSma1V7WNvtykjuABYEV2zWHnhYp0wrcjzWkwVAVix7PTn9YfNfRX84vDPky86u6vymkWpQdYeOVnLXNUiuABY881rndVIsw54bc2rnfRY99fVsdrXSgzGaHVJqc57+4868dbZla7DVBGAa8FatfRpQQ01+vfPoVXSv4tuX7NUy3uP18C4Qv0QitEn+bWVWq26nhoyQcH2KZWuR3ABcG3dzR00avbFFd8HW7XQRy88rVNjCzW9IE4tJ1+newZcpCdbpyhj+bk6o3q+Ck6oXel6TBUBuNau3yplf9FakmTSUzVm0gSlzrhera9ernBhoVprTsXuPqvXNtLf6nZV7Iyllb7bixEXANeaxu2STNmnz1aOilG76GideOlChQsLD2kbXbNY+0pjDnssUgQXANfCTkByyj4K2zhx1xHvnA+2aqFlpz+vrO1NXdUjuABYVefqQs0pitbaB3sqqknSAcfavbleW0IFin2srqsaBBcAV4KtWuj02t9VfF+au1kP9huiby8bq0aT9ihYryykdl3eU39L/Ea937rV9Z31LM4DcCW0ep1eHNRPS794XH845xTNfaaTam4uVbtJI3VOz0WavOQLBRTQc7t36LdNe+pEVf7+rZ8QXABcC61coxf2NNO/k2ZK98085HiHr69Sq1u2S9pspR7BBcCKKb07aOxVQ7SvRYk+OetRDfj4ZklSm/GFSp63RKUWaxFcAKwIbdmqJmO2SpJu0ilKKX+sjRfPemZxHoDvEFwAfIfgAuA7xnE8220IADzBiAuA7xBcAHyH4ALgOwQXAN/hBlQAEcsIDPPsat608EQTaVuCC1UmefTUiH/pc8YMjPiXGMcfpooAfIfgAuA7BBcA3yG4AFgX6t1ZH+Rm6ZPNi/RBbpZ2Tm2tQFp77bq8pwLx8a77Z3EegDXFA7ro4+efljRPp9w1StH5ZddjCpoH9MMp0rTRD2lIwS2q8fYcV3UILgBWOD1P1j+eflaSdPGac5TwYmbFsdqtWiht4hp1ff+PSnEZWhJTRQCW7LyrQOkx0qDvzlf+HxoecGxXeqLubpilZp/aqUVwAbBiVqfXtam0QIG7EuQsXFbxuomJUaubsxVQQNWnzLVSi6kiACvCCmt9aS1p9hJJUjC1jZbfVFvfnfukJGl6QU1rtQguANZ0itmn05YUSpK6xb2jPtULK/a0/tPiC9RUy478l48CU0UAVgz+bqjiTDXdVm+pbqu3VGdUz9dpd9yo0xZdIklqOC7OWi1GXADs6LtJZw69QVvTy8ZDyX/J1LZXivRd2huasDtZccu+t7ZFGcEFwJq4yXOUPPnn7787c7zCCuvJFWcoaWO2tTpMFQF4Yv1bHSVJ3eZdoaSh9kJLIrgAeOS97s9Iksz0BOt9E1wAPNEiKlZhhRVVaP/ZgwQXAE+EFdajP7ZXvXGZv974KBFcADzz/Lv9POmX4ALgiT//0F2J80Oe9M3tEAA88W16WNVl57OJBzOO49mmHQDgCaaKAHyH4ALgOwQXAN8huAD4DlcVAUQsIzDMs6t508ITI969vEqD62i2YJfYhh3A4TFVBOA7BBcA3yG4AFgX1ShRK5/pdsBO1iuf6Wavf2s9ATjumZgYrb23s564YLzOqJ6vEiegsMKamfaalCYNHtHVSh1GXACs2XBbupZePlZnVM+XJF21vq8ndQguAFaEpzfT4usflyRN3ttQg5v10LZeu3Rey1N19lXXa35RUP2+zbNSi+ACYMWHbaco2gRV6JTq7rcuksJlj7RxiooU/el8XfbJCN1Wd43WvX6y61oEFwArwgqrxAnpqrWDlfzXQ596mnL9XJU4IV3afp7rWgQXANeimiRVfL12YmvP6xFcAFxbOaq5JKnvjdcr8fFZv9h2dP3F2vvbHq7qEVwAXAvFhRVQQNWn/PITT6NNUAELsUNwAXDtpJNyFFb4V9uVOKGI2v0aggtAlVpfWqzq24pd9UFwAahS571wm4JfLHDVB8EFwLWCM7YooIAmbPhaoT6dD9tm3RsnqdfCi3XCPb+8eB8JgguAFV3mXabEYHVt6h17yLF9v+mut7qPU937Dz1WGQQXACua3BXW5H119d6VD2nVk90lSSY9VVtH9tIjDz+hdtUC0uwlVmrxdAgAVoSWrdBLZ/XRs8+F9fGgR3T+xts0/trH1Smm7CriWdkXqJrWW6lFcAGwpnRtjmIubqARnf6grBceU9v3b5QktXgnrJgvlsjWA+sJLgBWhbZtU/Sn2zS4SVel6OcbUm3ussEaFwDfMY7j2W5DAOAJRlwAfIfgAuA7BBcA3+GqIoCIZQSGebYoPi08MeKd6xlxAfAd34y4kkdPjTjpc8YMjDi5AfgPIy4Anvhk8yIlZtbypG+CC4BnXm7+lfKHdrfeL8EFwFObT7e/ckNwAfBUqz/Ott4nwQXAU6v/7W4rssMhuAD4DsEFwHcILgC+Q3AB8BSL8wAggguADxFcAHyH4ALgKe7jAuALXnw+cX8EFwDrWty+3NP+CS4AVuUP7a6Xm38lSTrtxus8uR3CNw8SBOAvA5LSFKc5nvTNiAuAVXGT52hAUpqnNQguAL5DcAHwHeM4nu02BACeYMQFwHcILgC+Q3AB8B2CC4DveHIDav369Z3k5GQvugaUlZW13XGcBv/t8zgeZQSGeXY1b1p4YsT7mHkSXMnJyZo/f74XXf/PSh49NeK2OWMGengmxz5jzPrZlku7AAAFwUlEQVT/9jngv4upIgDfIbgAWFd6ZroGZ+/QB7lZ+jB3gT7IzVL6wrDUraOV/vmQNQArVo7voovT50qS7m34nCQprLDazbhWDd+LUfybsyUttVKL4AJgxcqzn9WWUIGe2tFLKR9dp3qZ0ao3IVMnaqH1WgQXACvCCuupHb2U1SmgFHl7cY41LgBWdPjPKN3bcKFCfTp7XosRFwA7jBSQ0Y7UWNU16ZKkmPmrFNqzx3opgguAFWOGvKqwHM2+8zEFyidzvZcOU9HEVNWbkGm1FsEFwLWoZk01uEaW7t7aSe/ndJAzu44GX/S1bmn5mYbct0vh+xyddfm1ivo8y049K70AOK6VbtykQU3KpodJypYkZT0YUJZa6rnyNlvuiNHzE+bpzqtHuA4wFucBVIkmD87SnPxWum/8OO24uqervgguAFVm4p8HaOruND31l7HacE+vSvdDcAGoMtXfnavF5zbTnPxWWvT7xyrdD8EFoEqVbsrV2MV9Kq48VgbBBcCKLTf9+tQv2D5F+z5uqS9OfULnDrmy0rW4qgjAtR1X99TeHgVKfPzQY1HNmmr9JSdIksZf97hmF5yooffcprrzKn9vF8EFwIrlvcdr4bqwLsn8vYykxLp79EXHiQpogcJyFJBRq4+uU/t7vlfdTe5uSCW4ALhWb0Km2p15jSTppZ4T1C3G0bUbe6vdjGsU3lFNLSeXSJJSvpivUgv1CC4AVpx4adnja+7TTx+yzvPkkTYSi/MAfIjgAuA7xnHs7zZkjNkm6Wh2Yqkvabv1E6mcY+lcpGPrfI6Vc2nO9mTHN0+C66hPwpj5juN0+W+fh3RsnYt0bJ3PsXQuOL4xVQTgOwQXAN85VoLruV9vUmWOpXORjq3zOZbOBcexY2KNCwCOxrEy4gKAiBFcAHynSoPLGHOWMWaFMWa1MWb0YY7HGGPeLD8+xxiT7NF5NDPGfGGMyTbGLDPG/OEwbXobY3YbYxaV//mbF+eyX70cY8zS8lqH7KZpyowtf2+WGGM82bzOGNNmv3/zImPMHmPMzQe1qdL3BjhYlX1W0RgTlPSkpAxJmyTNM8a85zhO9n7Nrpa003GcVsaYiyQ9KOlCD06nVNKfHMdZYIyJl5RljJl20LlI0kzHcQZ5UP9I+jiOc6QbPM+W1Lr8T3dJT5f/1yrHcVZISpMqfma5kiYfpmlVvzdAhaoccXWTtNpxnLWO4xRLekPSeQe1OU/SS+Vfvy2przHG2D4Rx3G+dxxnQfnXeZKWS2piu45l50l62SkzW1IdY0xjj2v2lbTGcZyj+RQE4LmqDK4mkjbu9/0mHRoWFW0cxymVtFtSPS9Pqnw62knSnMMc7mmMWWyM+cgYk+rleUhyJH1qjMkyxlx7mOORvH+2XSTp9SMcq8r3BjjAcf1YG2NMTUmTJN3sOM7B+4QvUNln4vYaY86RNEVl0zSvnOo4Tq4xpqGkacaY7xzH+crDer/IGFNN0mBJdx7mcFW/N8ABqnLElSup2X7fNy1/7bBtjDFRkmpL2uHFyRhjolUWWq86jvPOwccdx9njOM7e8q8/lBRtjKnvxbmU18gt/+9Wla0pdTuoSSTvn01nS1rgOM6Wgw9U9XsDHKwqg2uepNbGmBbl/ze/SNJ7B7V5T9Lw8q8vkPS548EdsuXrZhMkLXcc55EjtGn00/qaMaabyt4rr0K0RvlFAhljakjqL+nbg5q9J+mK8quLPSTtdhzney/Op9zFOsI0sSrfG+Bwqmyq6DhOqTFmpKRPJAUlPe84zjJjzH2S5juO857KwuQVY8xqST+qLNy8cIqkyyUtNcYsKn/tz5JOKD/XZ1QWnNcbY0olFUi6yIsQLZcoaXJ5FkRJes1xnI+NMSP2O58PJZ0jabWkfElXeXQuP4VnhqTr9ntt/3OpyvcGOAQf+QHgO9w5D8B3CC4AvkNwAfAdgguA7xBcAHyH4ALgOwQXAN/5f/uSZlaWbVeFAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x576 with 32 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "NUM_STEPS = 100\n",
    "import threading\n",
    "\n",
    "def fit_loop():\n",
    "  return tf.contrib.tpu.repeat(NUM_STEPS, fit_batch_with_infeed)\n",
    "\n",
    "\n",
    "def _run_infeed(session, infeed_ops, x, y, images, labels):\n",
    "  for i in range(NUM_STEPS):\n",
    "    if i % 10 == 0:\n",
    "      print('Infeed %s' % i)\n",
    "    session.run(infeed_ops, {x: images, y: labels})\n",
    "\n",
    "losses = []\n",
    "def _run_outfeed(session, outfeed_ops):\n",
    "  # hack: store output in losses\n",
    "  for i in range(NUM_STEPS):\n",
    "    losses.append(session.run(outfeed_ops))\n",
    "    if i % 10 == 0:\n",
    "      print('Outfeed: %s %s' % (i, losses[-1]))\n",
    "      \n",
    "infeed_thread = threading.Thread(\n",
    "    target=lambda: _run_infeed(session, infeed_ops, \n",
    "                               image_batch, label_batch,\n",
    "                               x_train[:BATCH_SIZE], y_train[:BATCH_SIZE]))\n",
    "outfeed_thread = threading.Thread(\n",
    "    target=lambda: _run_outfeed(session, outfeed_ops))\n",
    "\n",
    "fit_with_infeed_loop = tf.contrib.tpu.batch_parallel(fit_loop, num_shards=8)\n",
    "\n",
    "session.run(tf.global_variables_initializer())\n",
    "\n",
    "infeed_thread.start()\n",
    "outfeed_thread.start()\n",
    "session.run(fit_with_infeed_loop)\n",
    "infeed_thread.join()\n",
    "outfeed_thread.join()\n",
    "\n",
    "[predictions] = session.run(predict_on_tpu, {\n",
    "    images: x_test[:16], labels: y_train[:16]\n",
    "})\n",
    "\n",
    "plot_predictions(x_test[:16], predictions)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "pjG7NEmWZW2Q"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "VrwyluKFsNTp"
   },
   "source": [
    "Copyright 2018 Google Inc. Licensed under the Apache License, Version 2.0 (the \"License\"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License"
   ]
  }
 ],
 "metadata": {
  "accelerator": "TPU",
  "colab": {
   "collapsed_sections": [],
   "name": "TPU Fundamentals - External",
   "provenance": [],
   "version": "0.3.2"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
